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DR. ROBERTS: I really do want to welcome today, Paul Ekman, he is 
actually a legend! That’s it. 
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EMOTIONAL SKILLS 

DR. EKMAN: I want to thank you for this opportunity to give this talk 
today. It’s mostly new ideas and some new findings. I also want to apologize to 
some of the other speakers, some of whose talks I’ve missed. I’m still 
recuperating from major surgery and I have to take breaks each day, so it’s 
unavoidable that I can’t be at all the talks, much to my regret. 

I want to begin by describing what the consensus is within psychology 
about what an emotion is and then I will discuss each of four skills. Klaus Scherer 
yesterday objected to the word “intelligence”, I will object to the word “emotional” 
because I’m not at all sure that the people concerned with emotional intelligence 
are actually focusing on emotion. I think most of them are not focusing on what 
I’ll describe as the general consensus among emotion theorists and researchers 
about what emotion is. Nevertheless, there are skills relevant to emotion that 
might be relevant to emotional intelligence. That is my reach for the topic today. 
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Almost 10 years ago Richard Davidson and I co-edited a book called, “The 
Nature of Emotion” in which we invited 24 emotion researchers and theorists to 
write their views on issues that we knew there was disagreement about. These 
are half of all the 12 questions we asked them to answer about emotion and as 
you can see, they represent not orthodoxy, but quite a different range of views. 
For each question we picked five or six people, who we knew pointedly 
disagreed, to write specifically on that question. Here are the rest of the 
questions. 

Why am I telling you this? Because in the epilogue to the book, Davidson 
and I while looking across all of this, identified what these people agreed about. 
First, it’s that emotions involve information processing. You heard Klaus Scherer 
talk about his approach to appraisal; we get along so well because he looks at 
what precedes emotion and I look at emotional responses. But today, I’m going 
to invade his area at least in the first part of my talk. Information processing 
involves an evaluation typically of some event that precedes the emotion, and 
that it involves expressive and physiological changes distinctive for each 
emotion. But, there was disagreement about both the number of emotions and 
whether it’s a universal phenomenon. 
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However, there was no disagreement about the memories, expectations, 
methods of coping, or that there is subjective experience. In terms of the signal, 
which is what I’ve spent most of my life studying, there is disagreement about 
how many emotions have a distinctive signal and secondly, whether the signal is 
the source. That is, can there be emotions that have no signals, are those still 
considered emotions? In addition, there was clear consensus concerning mood, 
namely that moods last a lot longer and they often, but not always, have no 
obvious social cause. 

Incidentally, when considering the history of the study of emotion, there is 
really a gap in the English speaking world between 1924, Floyd Alport’s chapter 
in emotion, and Solomon Ash’s chapter in the middle 50’s. Emotion research 
pretty much died; behaviorism killed it in America. Expression psychology in 
Europe and particularly in Germany was quite active until Hitler killed it when he 
had the social scientists who didn’t flee, give it a racist bias, so that you could 
identify inferior races from their expressions and gestures. So, there was a big 
gap. If you look at introductory textbooks in the 1950s, not only is emotion not a 
chapter, it’s not even in the index. It just didn’t exist; people didn’t get taught 
about it. “If I can’t measure it, it doesn’t exist”, has been sort of the motto of 
psychology. I think that motto is still around, but I think the tide changed in part 
because we can now measure at least the rear end if not the front end. 
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What about the disagreement about how many emotions are there? There 
seems to be general consensus that there are about seven, however, there is 
disagreement about love, hate, jealously and compassion. None of those do I 
consider an emotion. I would be glad to tell you why in the question period; I’m 
not going to spend time on it. That doesn’t mean they don’t exist, but not 
everything that is important or has feelings is an emotion. 

Is there a goal standard? Well, some have thought that subjective 
experience is the goal standard - if you are not aware of your emotion you don’t 
feel it. No one that has ever done clinical work believes that, I don’t think anyone 
who has been married believes that. Your spouse doesn’t always know they are 
feeling emotional and they act in a way that by all outside objective views is very 
emotional. Is physiology the gold standard? Well, I don’t know a physiological 
psychologist who doesn’t think it is the goal standard. Is expression the goal 
standard? Well, at one point I took that view, I no longer do, because I don’t think 
there is a gold standard. As Klaus said yesterday, there are many different 
characteristics that combine to distinguish emotion from related affected states; 
emotion is only one affective state. 
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Now let me turn to emotional skills. Most of what I’m going to talk about is 
based on a new book called “Emotions Revealed”, that was published last spring 
and is just about to come out in French, Spanish and Chinese. First, I should say 
that I don’t claim that the four emotional skills are the only ones, they are just four 
that I have considered. I think they are four important ones, but clearly there are 
other emotional skills. The first skill is to be aware of the emotion we are feeling, 
ideally as it is developing but at least in the first few moments before much or any 
action is taken or any words are spoken. This is not easy to achieve. I believe 
that this is so, because I don’t think we evolved in order for consciousness to 
muck around with our emotions. Again, as Klaus mentioned yesterday, our 
emotions save our lives because we don’t need to think about things first in order 
to become emotional. If, for example, you think of a near miss car accident, that 
very complex process of evaluating that car that is about to hit you and 
determining how you are going to manifest the behavior to cope with it, that all 
occurs prior to consciousness. You become conscious once you’re already doing 
it often after it’s already over, if you have survived. And if you had to think about 
it, you probably wouldn’t survive. And that’s when emotions really are in our 
service. Now, my conscious mind was involved in learning how to walk, but I 
didn’t think of one step I took today. When I learned how to eat, I was very 
conscious of each movement of that fork, but I didn’t think a bit about the 
movements of the eating implements at breakfast this morning. The fact that 
behavior is totally automatic without thought does not necessarily mean that it’s 
biologically built into us, but it does mean that it occurs without conscious 
deliberation or awareness. 
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Now, most emotion researchers agree that the appraisal process that 
turns on emotion often typically operates quickly outside of awareness and too 
fast for awareness to penetrate as it’s occurring. There are eight other ways, I’m 
not going to focus on them, but let me just put them on the board. These are 
different ways in which emotions can be brought about. Apart from the automatic 
appraisal which I think is the one that most often, both saves our life and gets us 
into trouble, and is the most common one. 

Let me turn away from that and come back to this first skill. Once we 
become emotional we are often gripped by it. We behave without consideration 
of how we should enact our emotion. It’s not that we think about, “Well should I 
challenge you on the insult you just gave me or should I just let it pass over?” 
That can happen, but it often it doesn’t happen. The first skill is to know that an 
emotion is occurring as early in the episode as possible. Sometimes our 
automatic appraisal is on target- it is a snake not a coiled rope, but sometimes it 
isn’t. Obviously, those who responded to things that looked snake-like probably 
put more genes into the gene pool than those who thought they were just 
passing a coiled rope and got bit. Sometimes we do misunderstand what’s 
occurring in our emotions. They are either too strong or they are inappropriate or 
they’re too weak. And this happens to anyone. And the antidote is to be aware 
that you are becoming or being emotional as early in the process as possible. 
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Now even the Dalai Lama, who I’ve had the good fortune to be able to 
meet with three times, does not claim that he can be aware of the automatic 
appraisal that initiates an emotional episode. But through hard work by 
developing skills that we don’t start out with, some people can learn impulse 
awareness. That’s the first skill, it’s impulse awareness - that means being aware 
of the impulse before the action, the impulse before the words are spoken. It’s 
very tough; I’m not convinced that everybody can learn that skill. Shortly, I’ll 
describe the kinds of people who are most equipped to be able to learn it and 
some of things you can do to enhance that skill. Now, just so you shouldn’t think 
that this is all coming out of Tibet and Buddhism, 45 years ago when I was still in 
the grip of clinical psychology, my psychotherapy supervisor said, “If you can 
increase the gap between impulse and action, you will have enormously 
benefited your patience.” That is a very common view in psychodynamics, 
psychotherapy. The Buddhist put it, “To recognize the spark before the flame”; 
it’s exactly the same concept. Now notice it’s not recognizing what gives birth to 
the impulse; what starts the spark. That, virtually no one thinks you can enter 
into. But to be aware and not necessarily let the flame occur, that’s the first 
extremely difficult, but extremely useful skill. 
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There are a lot of things you can do to try to enhance this, let me just 
mention a few of them. One is to keep a diary of regrettable, and just so you 
won’t feel too badly, non-regrettable emotional episodes. And then try to identify 
from the list of regrettable emotional episodes what’s the common appraisal that 
leads you to these regrettable episodes. You can usually, as Klaus and 
researchers have shown, figure out afterwards what it was that you’ve 
misappraised. What script did you bring in that doesn’t fit the situation, if you 
want to use that terminology. Typically, for laymen, it’s easier to think of it as a 
trigger than appraisal but the appraisal is making something into a trigger. In 
psychotherapeutic terms, one can work through the script and lay it to rest 
through the interaction of the therapist. My bet is that that never succeeds 
completely. The most one can do is to learn to recognize and weaken that 
particular appraisal process; much of emotional behavior and appraisal is 
learned. Once learned, is not unlearnable. I think of emotion as a fish trap, it’s 
easy to get in but hard to get out. People spend a lot of money trying to get 
things out of the system that have become automatic, I think you can weaken 
them but not eliminate them. 
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A totally different approach to achieving impulse awareness that I’ll 
describe later, is mindfulness meditation. The Buddhists have developed 
exercises in which you focus your attention exclusively on each breath you take 
and the sensation in your nostrils as the breath goes in and out. Or, on each step 
each you take. If you’ve ever seen people do walking meditation, they look like 
zombies because they are walking slowly. Or, eating meditation is really good for 
people who want to lose weight, because it takes you about five minutes to get 
each mouthful. There isn’t any hard scientific evidence at this point that 
mindfulness meditation improves emotional life, although there are many, many 
studies in which people who’ve done it claim in self-report measures that their 
emotional life has been improved. Up until very recently I couldn’t understand 
why focusing on breathing would benefit emotional life, yet I had seen so many 
people who acted in ways in which I could not act who had been meditators for 
10 or 12 years. 
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One of the things they shared in common was seeming to be aware of the 
emotions that were occurring much earlier in the interaction than I ever was. 
About a month before I wrote the notes for this presentation, like a bolt out of the 
blue, it struck me why mindfulness meditation might work. It is the very practice 
of learning to focus on an automatic process that requires no conscious 
monitoring that creates a capacity to be attentive, I think, to other automatic 
processes. We breathe without thinking; I presume none of you have thought 
about a breath you’ve taken since you’ve woken up this morning. You don’t need 
to, nature doesn’t require it. Think what life would be like if you have to think 
about each breath or you would die. And so, when you try to think about each 
breath without letting your consciousness drift to anything else, it’s extremely 
difficult. The first time I tried this, I could not manage. One of the techniques I 
used was to simply count and see how far I could get into the count before my 
mind drifted on to something else. And in a full day, I never got to sixty. The 
Buddhist teacher said, “That will take weeks and months to be able to have a full 
minute where all you’re doing is focusing on this automatic process, nature didn’t 
provide us the means to do that.” But it is possible to learn by a lot of practice 
and my suggestion is that when you learn that skill, or any skill, you are creating 
new neural pathways. You are creating neural pathways that nature didn’t give 
you, neural pathways for monitoring automatic behavior and they have the ability 
to transfer. So, if you can monitor your breathing you are going to be more aware 
of the automatic processes when that impulse is generated. 
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Talking to my buddy, Richie Davidson, who works on the brain and 
emotion, he thinks it makes sense and talking to the Dalai Lama a month ago, he 
thinks it makes sense; the Buddhists hadn’t considered before why it is helpful to 
emotions. The fact that people think something makes sense doesn’t mean it’s 
true, but it is at least suggestive. A research project is now going on at UCSF that 
I organized and is being led by Margaret Kennedy, in which very hard objective 
measures of emotion, not only self-report is being used, to look at before and 
after on the impact of mindfulness meditation. 

Here is the second skill. Not as good as the first, but the impulse has been 
transformed into actions and words. You are exercising, maybe not in the first 
instant but before it has gotten very far, how you are in that emotion. Again, 
emotions did not evolve, in my view, for you to be able to do this. It is not going to 
be easy for you to do this; it really requires learning a skill and learning a skill that 
is not like learning to ride a bicycle, bicycles were not part of our ancestral 
environment. And yet it is easy to learn a bicycle, both of my kids learned it in 
less than an hour. I hadn’t ridden a bicycle for 25 years, and I got right back on 
and rode a bicycle. Once you learn this novel piece of behavior, it sticks with you. 
The skills I’m talking about do not. They don’t connect to anything within our 
brain that allows them to be maintained, unlike riding a bicycle which connects to 
all kinds of things having to do with locomotion and visual motor coordination. 
What you want to be able to do is pause and consider “Is this the right emotion 
for this situation?” And am I acting it in a way that is going to be most useful to 
my goals and to my relationship with the other person? 
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Now, it’s not that you are unconscious when you’re emotional, you are just 
not conscious of being emotional. A memory researcher, I can’t remember his 
name, gives the example, which I think is an excellent one, of you’re reading a 
book and you’re turning the pages and after a few minutes you notice the fact 
that you’ve been thinking about the movie you saw last night and you didn’t read 
a word on any of the last five pages. When you were thinking about that movie 
you were not unconscious. You knew you were thinking about a movie- you were 
conscious, but you weren’t conscious of the fact that that’s what you were doing, 
that’s why you continued to turn the pages. That has a lot of applicability to 
emotional behavior. When we’re emotional, it’s not that we aren’t conscious, 
we’re very conscious, but we’re not conscious to the fact that we are being 
emotional. The Buddhists call that a “watcher”, there’s nothing watching you, 
there’s no meta-consciousness; there’s no attentiveness to the fact that you’re 
being emotional. 
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Now everyone reacts quickly in a near miss car accident but some of us 
react to all of life like if it is a near miss car accident. One of the components of 
what I call our emotional profile is how quickly we respond emotionally. Some of 
us have a very fast emotional response system, while for others the emotional 
response system seems to be slower. And in examining the latency of facial 
expression to emotionally arousing films, I have found that the speed of response 
is a general, rather than an emotion specific, phenomenon. Our language 
distinguishes such differences in latency for anger - a short fuse versus 
smoldering. But what we found in research, and I’ll shortly summarize, is if you 
have a short fuse for anger you have a short fuse for fear, sadness, and for 
disgust. It doesn’t matter how the emotion is elicited in terms of the circumstance 
or what the emotion is; most people show consistency in emotional latency. 

There is another reason why this skill is very hard to acquire and I’m proposing 
that when we become emotional there is initially a refractory period. And during 
that refractory period, information that contradicts the emotional feeling is not 
accessible to us, either from the environment or stored in our own knowledge 
banks. All that we filter is things that support the emotion; it’s a focusing. It’s 
great if it lasts a couple of seconds but a disaster if it lasts for minutes, because 
nothing that can disconfirm it can get into your consciousness. 
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Now what can we do to obtain emotional behavior awareness, skill 
number two? In my book, I describe exercises that we are now testing in the 
project I mentioned earlier, exercises in which you self-induce emotion using one 
of three techniques that have been our favorites in laboratory studies. Then, you 
focus your consciousness on the physiological changes in your body. So that 
those become a louder signal that tells you, “I’m starting to get emotional.” 
They’re there and some of them are capable of being reported to consciousness, 
but for most of us it is pretty dim, so the idea is to heighten it. The second 
technique is the identification of a regrettable emotional episode. And the third 
technique is heightening your sensitivity to the emotional reactions of others. In 
other words, I don’t get to see my face, but you do, so you have more information 
in some sense, particularly if I’m not speaking. So, you get information about 
what is going on from me from my face, and I don’t; there is very little facial 
feedback. We have experimental evidence to support the idea that most people 
can’t tell what their face is doing. You get great feedback for temperature and 
touch, but not for muscular movement. But from the other person’s reactions, 
they can inform you of what’s going on in you, if you become more sensitive. So, 
it’s not simply that they’ll enable you to better deal with the other person; they 
enable you to acquire this skill, to realize you are becoming emotional. Not 
everyone will benefit equally from these exercises. I mentioned one reason a few 
moments ago, which is whether your emotional profile responds slowly or 
quickly. Those who respond more slowly have a better chance of being able to 
acquire and use this skill. There is a second feature of emotional profile which 
enters in, which is the magnitude of your emotional response. 
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In collaborative research with Bob Levinson, for the last 10 years we have 
been slowly analyzing a data set that involves three response systems - the 
nervous system activity, facial muscular movements and subjective experience; 
three emotions- fear, sadness, and disgust; and two different emotional eliciting 
circumstances- viewing short film clips and reliving a past emotional experience. 
The reason we’re limiting it to these three is because we have really not been 
able to find films for other emotions that are emotion specific. It is essentially a 
2X3X3 design, but don’t expect to see the ANOVA; the data analysis will be 
complete this winter. So we have two circumstances, films and memory, with 
three emotions with three kinds of emotional responses. And we found generality 
across responses, across emotions, and across eliciting circumstances in this 
strength of emotional response. And I know unpleasant things were said about 
factor analysis but when you do a factor analysis we found that when you put the 
facial measures, the autonomic and the subjective in, they all load on the same 
factors. Another way of putting that more simply is if you just look at straight 
means or you look at intercorrelations. You can take an aggregate measure of 
autonomic activity that correlations with an aggregate measure of the strength of 
facial muscular contractions which correlates pretty highly with ratings on the 
good scales. So, the strength of our emotional response is something that seems 
to be for most people. 
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Another thing that is really quite interesting is that people are quite aware 
of this characteristic and a simple questionnaire that asks people to rate the 
strength of a typical emotional experience in their daily life, given to them three 
weeks before they came into our laboratory, predicts better than .4 the strength 
of facial autonomic responses. It doesn’t predict the self-report very well, but it 
does report physiological and expressive behavior across these three emotions 
and across these two tests. So if you are a big responder, you know it, people 
are letting you know. And if you are a very small responder then probably your 
spouse is saying, “Why don’t you seen to care about things the same way I do.” 
Another really amazing thing is that the people who study marital interactions 
have found that people who have different emotional profiles often marry each 
other. They then spend a lot of time trying to convince the other person, you 
should have the profile I have, it’s the right one and yours is the wrong one. 
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So if we look at this, the strength of response and the speed of response, 
it turns out that all four cells are filled. That is, there are people who are slow, 
moderate responders; I said that they should make good negotiators or 
mediators, which is the hypothesis. What is not a hypothesis is that those people 
exist. Furthermore, there are people that are fast, extreme responders; these are 
the attack dogs of life. There are uses for attack dogs but attack dogs don’t work 
well as negotiators or mediators. Is this modifiable? Nobody knows. My bet is 
that this is not modifiable except through trauma. And what trauma does is move 
everybody into cell four. Now, it could be that if we were to design traumatic 
experiences of ecstasy, that is traumatic in that they are enduring, they last for 
many, many hours and they are spectacularly overwhelmingly enjoyable, if we 
could design such environments, maybe we could make other shifts. But one 
way of thinking of PTSD, is it moves people into cell four. 
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The third emotional skill is becoming more sensitive to how others feel. I 
mentioned two good reasons for doing that. One, is so you better understand the 
other person and can better calibrate your own responses. The second, is so you 
better understand what is going on in you. I am going to emphasize the face, but 
I do believe that voice is every bit as important as the face; it’s different in 
interesting ways. I just stopped the voice, I pause it, no signal. I can’t stop the 
face, it’s always there. The voice reaches people who aren’t looking at you. Can 
you imagine what it would be like to do child care if there weren’t vocal signals? 
You would always have to be looking at your kid, it would be impossible, you 
would be totally preoccupied. But hunters can’t communicate with their voice 
because they would give away their location to the prey. So, both of these are 
overlapping and highly redundant. It’s only during very explicit deception 
situations that we have found disconjunctions; they are highly overlapping and 
consistent. But we are now in the position that we can really teach people about 
facial recognition sensitivity. 
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So let me show you, what we’re doing. Seven emotions, grouped into five 
and two, because the five there has the strongest evidence and for the two, there 
is still some argument about. I don’t think there is much argument, but I have to 
acknowledge that some others do and we are just resubmitting our fourth rebuttal 
of another article that says that contempt doesn’t have a unique signal. Actually, 
the evidence in Indonesia is the best in the world, what an unexpected finding, 
but I think just because of the linguistic representation. When you use a story, 
which face fits with a person who claims to have accomplished things that other 
people did not, then you get extremely high agreement across cultures in 
identifying what we said is the contempt face. But the word itself, particularly in 
English, can you think of a worse word than contempt? Our research and that of 
many others have shown that people worldwide have no problem and need no 
aid in interpreting a facial expression of emotion, if it is seen in isolation and it is 
one of these seven. If it is seen with accompanying speech and it doesn’t 
contradict this speech content, then no one needs tutoring. But life isn’t often like 
that. 
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Incidentally, Darwin proposed on the basis of one data point, that the 
ability to recognize emotion was innate, not just the expression. His one data 
point was his son William, his first of 10 children. And Darwin’s two keys to 
success were he never had a job in his life and he never gave an academic 
lecture. He observed and kept a daily diary on each and every one of his 10 
children and he did experiments with them. When William was between two and 
three, and William had never seen another child at that point, Darwin asked the 
nanny to put a look of distress on her face and to show a few sobbing 
movements. William immediately went and put his arm around her to comfort her. 
So, Darwin said he’d never seen this behavior before, yet he knows what it is. He 
knows what it is, not just because he was able to give us a rating on 
pleasantness and unpleasantness or choose sadness from one of the lists, but 
because he showed the appropriate behavior toward someone who was sad. 
When expression contradicts speech content then our evidence, based only on 
studies in English speaking countries, is that people don’t use the information in 
the face, instead they emphasize speech content. Even if you tell people that the 
people they are about to see may be lying to them, they still emphasize mostly 
speech content and ignore the valuable information in the face and the voice that 
could be informative to them when particularly looking at discrepancies. 
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So, we’ve developed two self-instructional CDs to improve emotional 
skills. One is to teach people how to recognize micro-facial expressions. In 1969 
we published our first article about micro-facial expressions and so did Haggard 
and Isaacs. We said that they are the consequence of deliberate concealment 
and Haggard and Isaacs said they are the consequence only of repression. I 
know we’re right because we have a lot of experimental data and at least from 
looking at clinical interviews from psychoanalysts, I have been able to confirm at 
least three or four examples. There is every reason to think that the person is 
totally unconscious and you see the micro expression. You can’t tell from the 
micro expression then whether the concealment is conscious or unconscious, but 
you can tell that an emotion is being concealed. 

I am going to show you an example of one so you won’t think this is 
something I made up in the laboratory, and I’ve chosen Kato Kalen. The 
Europeans here may not know who this famous person is, he was sort of a 
sycophant for O.J. Simpson, he testified in the trial. Kato had to testify because 
he was not on trial and Marsh Clark, the prosecutor, was badgering him and at 
this point had caught him in a lie. He had said he had not negotiated a book 
advance but she knew he had from the book publisher. And this later came out, 
and she’s pushing him because she’s trying to destroy his total credibility to the 
jury. 
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MR. CLARK: Mr. Kalen you got a lot of money for your appearance on 
Current Affair didn’t you?” 

MR. KALEN: Um, yes. 

MR. CLARK: And as a matter of fact, isn’t it true Mr. Kalen that you have 
a book proposal out for about half a million dollars right now don’t you? 

MR. KALEN: No. 

MR. CLARK: You don’t have a book proposal? 

MR. KALEN: No. 

MR. MARSH: Aren’t you represented by the William Morris Agency to 
write a book? 

MR. KALEN: No. 

MR. MARSH: Haven’t you written a book proposal that has been 
submitted to St. Martins for publication? 

MR. KALEN: No. 

MR. MARSH: You are not aware of any contract for half a million dollars 
that you have signed. 

MR. KALEN: I know it’s out there but I haven’t done that. 

MR. MARSH: You haven’t signed it yet? 
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MR. KALEN: Oh no, I heard a story about that, that’s not true. 

MR. MARSH: You don’t have any book proposals out? 

MR. KALEN: No. Don’t want to do a book. 

MR. MARSH: Do you plan to write a book in the future Mr. Kalen? 

MR. KALEN: As of today, no way. 

DR. EKMAN: Incidentally, just to show you that when we teach people 
about detecting deception from demeanor, we don’t emphasize just the face, of 
course. He showed what we just call soft voice and it takes some skill on the part 
of the interviewer to get them into a position where they either have to lie or tell 
the truth. Some people, not everyone, do a soft voice, just as he did. 

Okay, so now I’m going to show you the same thing again but first you’ll 
get a normal speed then slow-motion version of the micro expression that was 
contained in that segment. Notice that like all of the micro expressions that we’ve 
identified it is very fast; this one is about 1 /20 th of a second, it is very extreme. If it 
was on the face this long, no one would have trouble but the problem is that it 
isn’t. 
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Now the question is can you teach people to recognize it? We know from 
our research that nearly everybody misses them. Danny Goleman put in his 
book, “Destructive Emotions” our finding three years ago, which we haven’t 
published, that two Buddhist monks did spectacularly well in identifying micro 
expressions and the Buddhist world is very excited about that. They were a little 
discouraged to hear that we can now teach anybody in 25 minutes to do just as 
well. 


This is the micro expression training tool that we developed and I’ll show 
you how it works and then some results from it. Now this tool has a number of 
parts. It has a pretest in which you see 14 micro expressions and it computes 
your accuracy score, so you know where you are to begin with. Then, there is a 
training segment, which I’m going to show you an example from where emotions 
that are often confused with each other are shown in a morphed version with 
commentary. Then, there is a practice where you get to give yourself feedback, 
make judgments and find out whether you’re right or wrong. And then there is a 
review that again, contrasts emotions which are easily confused. And finally, 
there is a posttest. So, I’m skipping the pretest because of time. Now, here is the 
first of the training segments: 
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“Anger and disgust are sometimes confused with each other but you can 
see they are very different. Although the brow comes down in both, watch what 
happens in the eyes. On the left, in anger, the eyes are glaring, they are 
narrowed. And on the right, not glaring. Also, watch in the next time it cycles 
around, the lips. See how tightly pressed they are? But they are relaxed in 
disgust. All of the action in disgust is in the center line.” 

Now, you could say, why am I using such extreme expressions? Because 
that’s what you see in micro expressions, they are typically just as strong as this, 
as you saw in Kato Kalen. So this is fear and surprise, this is sadness and fear, 
this is happiness and content. And after you have gone through that, and you can 
see this is all set up so it’s self instructional, you don’t need a teacher. We used 
to teach this, now we don’t, we just give people the CD. 

The practice session works like this, let me just show you an item or two 
from it. There are 28 faces, and incidentally throughout this you never see the 
same person twice. You now you have to say, what emotion was that? So let’s 
suppose you say you thought she was angry, and you were wrong, you get feed 
back. You can try disgust; it tells you that you were right. So, you get to do that 
28 times then you get a review, this whole training takes about 24 minutes. We 
get about a 40% increase from pre to post accuracy. 
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In collaboration with Mark Frank at Rutgers we collected data on Coast 
Guard criminal investigators, and randomly assigned them to a training or a 
control group. The training group got the training that we just covered, the control 
group did not. And then we showed them eight videotaped clips. Kato Kalen was 
one of them, Kim Philby, the notorious British Spy was another, Kathy Webb who 
lied about being raped was another; all the great liars of CNN. In each of them 
the person is talking and there is a micro expression embedded, and afterwards 
the person is asked to rate the emotions they saw. And there is the training 
versus control group, it’s a whopping effect. Now, we didn’t get them perfect in 20 
minutes. So the study that is now underway is doubling the training time, we think 
we will move them up quite a bit. 
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There are also subtle expressions which are every bit as important but 
different. They are little tiny expressions. They are evident when emotion first 
begins, very often before the person knows how they are feeling. They are also 
signs of leakage as a person is deliberately concealing, and instead of getting a 
micro expression, they wiped everything off of their face expect a fragment, and 
that is what you see. Let me show you the subtle expression training tool, it is set 
up in a slightly different way. It uses only one person rather than 56 different 
people, the reason for that is because in order to see subtle expressions you 
have to hold the physical features constant. As it’s coming up I will tell you that 
there is some evidence in terms of being able to voluntarily control your facial 
muscles and doing what I’m doing, which less than 5% of the population can do, 
but if one identical twin can so can the other, but not so for fraternal twins. For 
each emotion you get to see the subtle expressions and you get to pick the 
speed at which you will get to see it. And you work your way through and see 
these for each of the emotions. Then, when you’re done, you go to practice and 
in practice, all of these subtle expressions from all the emotions come at you in a 
random order. Each time you take it it’s a different random order and you choose 
the speed and you get the feedback. So I’ll give you a reasonably fast speed and 
at the end you find out your percentage correct and people work at this, without 
being encouraged to, in order to get up to 100% which people get within an hour. 
So suppose you call that fear and you’re wrong, it just gives it to you again. And 
then you click to the next one, and you go through them all and then you get your 
percentage correct. 
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Let’s just compare the micro expressions with the subtle expressions. In 
terms of the difference in speed, micros are always fast, whereas subtle may or 
may not be fast. Intensity of micro is extreme, in subtle it’s slight. In micros it 
involves the whole face, in subtle one region. If you’re interested in playing with 
these, either to get better or to try them, I put these out on the internet site in 
June and now there are seven teams working with them with Asbergers. There 
are four psychiatry departments using them in training. The Foreign Service 
Institute is using them in training all Foreign Service officers. So, they are getting 
used. 


I want to turn now to the last of the four skills. Using the information about 
how others are feelings constructively. The METSET empowers you to get 
information that in Irving Goffman’s terms, you weren’t given. I know that Irving 
would say you’re stealing information from people. So, it becomes important how 
you use that in a way that it’s not going to make people feel you’re invading their 
privacy, which indeed you are. And that you’re doing it in a way that will be 
helpful rather than exploitive. 
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First, never presume you know what’s causing the emotion. Emotions 
don’t tell you their source. You can pretty much guess from the context that Kato 
Kalen is scornful and angry at Marsha Clark, but maybe he’s thinking about a 
fight with his wife that morning, or maybe he’s angry at himself for starting a lie 
that now he got caught in. The first thing to find out is what is the emotion’s 
source? Usually, it’s within the social context, but don’t be too egocentric and 
always presume, “I am the source.” In my book, I did examples of what to 
consider in family life, in the workplace and in friendships, so they are very 
different considerations, I believe with different entitlements. I have a good 
enough relationship with my grown children that I feel entitled to say to them, 
“Did you have a bad day, did something that just happened annoy you? I don’t 
feel entitled to that with my wife, I think it’s a different relationship with your 
spouse than with your kids, but it depends on your spouse and it depends on 
your kids. 
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For each emotion, I give three different contexts to consider in how you 
use this information. I just want to emphasize some general guidelines. Often the 
best course is to say nothing about what you’ve seen but just be alert to the 
possibilities, or you might say is there anything more you want to say about how 
you are feeling? Or, you might say I had the impression you were just feeling 
something more than what you told me about. You might be more specific, 
asking about the emotion you spotted. But how you respond depends on the 
nature of relationship; its past history and intended future and your knowledge of 
that person. You may not always be entitled to comment, even vaguely on the 
emotion you detected. Although, I don’t believe that relationships work better 
when people understand and acknowledge how each other feel, that isn’t always 
so. 


As I emphasized earlier, some skills are hard to acquire and some are 
easy. The first two I described are hard to acquire and as best I can determine 
they require at least weekly practice, probably daily practice for maintenance. 

The Dalai Lama told me that if he is traveling and he is not able to mediate for 
three days he loses his impulse awareness. And incidentally, when I was a 
patient in psychoanalysis way back when I believed in that, I went four hours a 
week and I thought, “Boy that is a lot of time invested in behavior change.” Four 
hours a week? These Buddhists do 70 hours a week, 80 hours a week, year after 
year after year. They are the Olympic athletics of the mind who have resculpted 
their brains. 
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So, I’ve described four emotional skills. The first two I think are difficult to 
acquire and require daily practice. The next two I think are very easy to acquire 
and I think that you rapidly reach asymptote; you don’t need a lot of practice. But 
skills without knowledge aren’t enough to improve emotional life. You have to 
understand each emotion - its storyline, the universal themes that trigger that 
emotion, some of the more common variations on those themes, the appraisals 
that are commonly involved, the function of the emotion and what it does for us 
and how it relates to moods and when and how it becomes involved in emotional 
disorders. It’s a tall order, those are all issues I addressed but I certainly haven’t 
resolved in my book, but they are all things that we need to consider. Again, I 
want to emphasize that I’ve only described the four goals that I’ve either 
developed tools for or have been thinking a lot about. The others I think are all 
relevant, to be leading a skillful emotional life, one that is constructive to one’s 
self and others. Thank you very much. 
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DR. ROBERTS: Thank you very much, questions? 

DR. GADE: Do you find that each emotion is treated differently in terms of 
these four skills? For example, in my own introspection I think I’m pretty good at 
becoming aware when I’m becoming emotional in an angry way but probably not 
in a sad way. 

DR. EKMAN: I have no data on that. Since individuals differ in almost 
everything that’s important in their life, I suspect that there are individual 
differences, for example, in which emotions are most accessible to 
consciousness and which are not. But we haven’t studied that, nobody has 
studied that that I know of. But if I were going to do research on that, which I’m 
not, that would be my bet. On the other hand, the Buddhist’s report that these 
exercises and mindfulness is only one of them, benefits all emotions. But that’s 
their report; I mean they report other things that I know are totally wrong, not 
because they want to be wrong. We all report our own experience which is in 
itself idiosyncratic. I don’t know, it’s a great question it could keep a whole 
laboratory busy for many years, if it were to be started. 

DR. KYLLONEN: Has the method of lie detection that you talked about 
been used in the court system? 
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DR. EKMAN: I’ve only appeared as an expert witness once. This was a 
guy that allegedly murdered his wife, by riding back and forth over her in her own 
car. When they came to arrest him, he was reading my book, “Telling Lies.” And 
so all they wanted me to do was to testify about what was in the book. I don’t like 
how psychologists or psychiatrists participate in the judicial system in the United 
States, I think it’s a very non-scientific way the jury can’t find out how many other 
experts you’ve seen before this one, so I won’t do it. I will work with law 
enforcement, I’ll work giving them advice about interviews they have conducted 
or are about to conduct, and I do give training to law enforcement and judges. I 
have rewritten federal instructions to jury’s about how to evaluate credibility, but I 
will not appear as an expert in a court unless we a get a system where the jury 
gets to know how many experts were consulted and what was their consensus. 
Which incidentally, occurs in the U.K; the U.K. is so far ahead of us in the 
criminal justice world it’s shameful, for us, not for them. 

DR. MATTHEWS: You talk about four different kinds of competencies or 
skills, but if people don’t support some general intelligence they would have to be 
intercorrelated. So, do you have a sense to follow these different areas of 
competence to see if they are correlated or if they are independent? 
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DR. EKMAN: I think the first two are but I don’t really know if they are 
correlated with the last two. This is really amazing to me, that something that is 
so valuable as recognizing subtle and micro expressions, people don’t know. And 
even experienced policemen, don’t know, and experienced lawyers; they just 
don’t get it, they don’t see it. But yet, they can learn it so fast. So, I’m delighted 
actually about both, because if they already knew it then what would I have to 
teach them? And if they couldn’t learn it then again, what would I have to teach 
them? I’ve written an article called, “Why don’t we catch liars?” It offers five 
different explanations about why we don’t already have these skills which are 
acquirable. As I mentioned, the before and after studies are only just now starting 
to demonstrate this ... 

DR. ZEIDNER: You seem to take issue with the emotional component of 
emotional intelligence but when I try to map your four skills into some of the 
current models, whether it be the Salovey, Mayer, or the recent Goleman 
orientation, which leaves sort of a four cell, two-dimensional space. But it’s 
basically the same factors. The way you go about perhaps breaking them down 
or specifying their operationalization may differ; it may be a bit more constricted 
than some of the others, but basically they’re also talking about identification and 
awareness of emotion in yourself and others. If you’re basically using the same 
terms, why do you have problems with the emotional component of emotional 
intelligence? 
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DR. EKMAN: I would not claim expertise on the emotional intelligence 
literature, just a bit of knowledge of it. But my limited impression is they’re talking 
about things that are more like enduring traits of how emotions are manifest in an 
interpersonal relationship, than with more momentary emotional states. Not that 
the two are unrelated, I think they are related. I wouldn’t be willing to push that 
very far, maybe if this is relevant to emotional intelligence, good. It makes me 
feel more legitimate for being here. 

DR. ZEIDNER: One other question has to do with the positive manifold 
you found across various channels in terms of latency. Could this be due to the 
negative affectivity factor? 
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DR. EKMAN: There are different signals and autonomic patterns I know 
people like John Cacioppo have a strong belief that that doesn’t exist, but 
Levinson and I, and I could name you eight or nine other investigators, have 
strong multicultural evidence that there is a different autonomic signature for 
these different negative emotions. I don’t think any emotion is negative, I think 
some emotional episodes are negative in a sense that they are destructive for 
the other person and for you. But, an enormous number of people enjoy being 
disgusted -just look at your kids! And you may think that sadness is a toxic 
emotion but look at the number of people who go to see movies and tell each 
other about novels that will make you cry. So, there is no question in my mind 
that the positive-negative distinction is a gloss that does harm to our 
understanding of the world of emotion and to actually how people behave; they 
don’t behave in terms of negatives. The emotions have a lot of commonality but 
there are running in different musculatures, in different physiologies. Any 
cardiologist will tell you that individuals differ in their cardiac activities, and those 
are differences are apparent fairly early. So yes, you’re going to find some 
consistency. For example, fear and anger are both heart rate accelerators, but 
the hands get hot in anger and cold in fear. So there are some commonalities but 
some differences, that’s why we used a global autonomic measure to look at 
magnitude of response. I am a sharpener and not a lumper and I don’t think we 
should lump it until after we have made the distinctions and gotten the evidence 
to show these distinctions don’t get us anywhere. Rather, we should start with 
that kind approach, and you never find out what you’re losing. 
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DR. LIVELY: I really like your example about Darwin’s son. It seems as 
though there might be a contradiction in that and what you’re saying about all the 
policemen and firemen and people who don’t know how to recognize and 
supposedly then can’t respond to emotions. So, I wonder if there is some 
developmental process? 

DR. EKMAN: No, I think you missed an earlier point and I probably went 
over it too fast. When people are not trying to conceal or inhibit their emotions, 
you don’t need to teach anybody, everybody gets it, we have evidence for that. 
What you need to teach people is to recognize subtle and micro expressions, 
without a doubt that is what most people miss. When people ask me what’s the 
one book I should read I say Darwin’s “Expression of Emotion” book. Even 
though I have 15 books out there, read Darwin’s; it’s the best book out there. But 
thank goodness, he left a few things alone, he only has two sentences in that 
book about deception; he left that for me. So, he didn’t really consider the issue 
of concealment or deception. 

DR. LIVELY: Do you think some people have it without knowing 
consciously they have it, and if so how do they get it? 

DR. EKMAN: Maureen O’Sullivan is going to report on that after the 

break. 
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PARTICIPANT: I have a question regarding emotional profile. You say 
that trauma can move anyone into cell four, but do you know why people start out 
where they do? Do you think it’s genetic, or is there something during people’s 
childhood that cause people to start off being in any of these cells? 

DR. EKMAN: Nobody has the answer to that question; I’m not the only 
one who can’t give you the answer. These days, the rage and neuroscience is 
plasticity and I get into big arguments with my friend Richie Davidson, who is a 
big promoter of plasticity based on evidence largely from the motor system. Is 
there as much plasticity for emotion? I think we have to watch out that we don’t 
become Lockeans again, which is what I think the neuroscientists are doing. So, 
we don’t really know. In the whole temperament tradition, which really hasn’t 
looked at specific emotions, says there is 50% inheritability. I think it is really the 
best evidence around. So, nothing is all the product of one thing, it’s all 
interactive. Experience does change, but I believe in terms of the emotion 
system, there are constraints on changeability and that you have to really wallop 
people hard in order to change their profile. However, I can’t conceive of a 
human subjects committee that would allow me to wallop them, even if I wanted 
to do the ecstasy thing I could never get that though my human subjects 
committee. 
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DR. KYLLONEN: I’m very excited about the next session. One of the 
strengths of this program is the diversity of perspectives and the diversity of 
approaches and findings, and we certainly are going to see that this morning. So 
what I would like to do is start off by introducing Professor Lawrence James from 
the University of Tennessee. He is going to talk about the Conditional Reasoning 
Approach to Personality Assessment. 
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THE CONDITIONAL REASONING APPROACH TO PERSONALITY 
ASSESSMENT 

DR. JAMES: Let me thank Richard for the opportunity to address this 
audience, and also for scheduling me right after a legend. I’m not really going to 
talk about emotions today and only tangentially about intelligence. Rather, I’d like 
to address the issue of measurement systems of personality and a new system 
we’ve been working with. 
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Objectives 

1. Introduce a new system for personality measurement: 
Conditional Reasoning Test for Aggression 

2. Discuss how system enhances our ability to predict which 
individuals will: 

Aengage in physical assault 
Abe verbally abusive 

Aquit, often in a disruptive manner (e.g., without notice) 
Abe habitually absent and/or tardy 
A lie 
A steal 

A perform poorly 
Abe undependable 


And the objectives are to introduce a new system for personality 
measurement, referred to as the Conditional Reasoning Test for Aggression. 
Aggression will be our personality construct of interest and I will discuss how this 
system enhances our ability to predict whether individuals will engage in: 
physical assault, verbal abuse, lying, stealing, or poor performance. Our 
hypothesis is that many acts of undependable behavior or deviance are products 
of indirect aggression or passive aggression intended to seek retribution for 
perceived injustice or victimization. My background is industrial psychology, so a 
good part of our work is done in the field in industrial situations. 
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Background 

Everyday Reasoning: 

Many waking hours are spent thinking inductively. 

For example: 

• What are the pros and cons of awarding prescription 
privileges to psychologists? 

• What strategies might terrorists use in the future? 

• Which is more conducive to education, public schools or 
private schools? 

• What is the relative effect of diet on health? 


I’d like to start with the basic idea of what is the type of reasoning that we 
are addressing. We spend many waking hours thinking inductively and we could 
ask, “What are the pros and cons of awarding prescription privileges to 
psychologists?” “What strategies might terrorists use in the future?” “Which is 
more conducive to education, public schools or private schools?” And, “What is 
the relative effect of diet on health?” One assesses which evidence is credible, 
which assumptions are tenable, which arguments are valid, and ultimately which 
conclusion or perhaps set of conclusions, which don’t necessarily have to agree, 
might be true. This is inductive reasoning, although it focuses less on formal 
problem solving than on the deliberations that people use everyday to attempt to 
make sense of their environment or to decide upon reasonable ways to adapt to 
those environments. 
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A significant portion of everyday reasoning attempts to 
determine what a reasonable person would do when faced 
with frustrating or threatening circumstances. 

• What would a reasonable person do upon learning that 
she or he has been passed over for promotion? 

• How would a reasonable person react to being kept 
awake at night by the barking of a neighbor’s dog? 

• What is a reasonable response to being hit from behind 
while sitting in a car at a stoplight? 

• What is a reasonable amount of time to wait in an 
examining room for a physician to appear? 


A significant portion of everyday reasoning attempts to determine what a 
reasonable person would do when faced with frustrating or threatening 
circumstances. What would a reasonable person do upon learning that he or she 
has been passed over for promotion? Or, how would a reasonable person react 
to being kept awake at night by the barking of a neighbor’s dog? Or, what is a 
reasonable amount of time to wait in an examining room for a physician to 
appear? 
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Individual Differences 

People will differ in what they consider reasonable 
responses in each of these situations. 

Reasoning of normal people is bounded by the cultural 
norms (i.e., standards) and cultural values (i.e., ideologies 
and rationales that support the norms) that define prosocial 
or socially adaptive behavior in our society. 


Our concern is with individual differences; people will differ in what they 
consider to be reasonable responses in each of these situations. Basically, 
however, the inferences that pro-social people make about reasonable behavior 
are shaped by having been socialized into a culture, which is to say, having 
internalized the culture’s norms, ideologies and rationales. Inference is consistent 
with cultural norms, whereas ideologies and rationales tend to be taken as 
reasonable, sound and sensible. 
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Justification of Aggressive Behavior 

With this cognitive mindset, prosocial individuals typically 
find it difficult to understand why people engage in 
aggressive, antisocial behavior. 

• Why would someone shoot their supervisor or neighbor 
(or neighbor’s dog)? 

• Why would someone steal major items from their 
company? 

• Why would some students vandalize their own schools? 

• Why do some people continuously engage in spreading 
malicious gossip? 

• Why do some people try to control and dominate others? 

• Why are some people habitually absent or tardy? 

• How do some people justify deliberately inferior 6 

performance? _ 


Let’s then talk about justification of aggressive behavior. With this 
cognitive mindset, pro-social individuals typically find it difficult to understand why 
people engage in aggressive or antisocial behavior. Nothing in their cognitive 
repertoire provides them with a balance to explain or prepares them to answer 
questions like “Why would someone shoot their supervisor or their neighbor or 
their neighbor’s dog?” Or, “Why would someone steal items from their 
company?” How do people justify deliberately inferior behavior? All the pro-social 
people can say about dysfunctional behaviors or social deviant behaviors is that 
they are socially deviant, according to the standards of the culture, and these 
people who participate or engage in these types of behaviors tend to be irrational 
or foolish. 
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How Aggressive People Think 

Example: As a result of being passed over for promotion, the 
supervisor is physically attacked. People who attack their 
supervisors often regard their aggression as a justifiable act of 
retaliation for having been persecuted and victimized. 

Chain of logic that leads to this justification typically begins 
with how supervisors are FRAMED. Future aggressors frame 
interactions with their supervisors as dominance contests in 
which the bosses use their institutional power to force the 
subordinates into submissiveness. 

Supervisors are thus framed as “intimidators” and 
“persecutors.” 


Now, pro-social people then oftentimes are mystified to learn that people 
who perpetrate aggressive behaviors often think of their actions as being 
reasonable. Now, how do aggressive people think? And let’s use as an 
illustration here, a result of a person who has been passed over for promotion 
and subsequently attacks their supervisor. People who attack their supervisor 
often regard their aggression as a justifiable act of retaliation for having been 
persecuted or victimized. The chain of logic that leads to this justification typically 
begins with how the supervisor, who’s been the target of aggression, is framed. 
Future aggressors tend to frame their interactions with supervisors as dominance 
contests, in which bosses use their institutional power to force the subordinates 
into submissiveness. Supervisors are thus framed as intimidators or perhaps 
ever persecutors. One of the things I want to do is to emphasize how important 
differences in framing are, and I’m not so much concerned about quantitative 
differences as I am concerned with qualitative differences. 
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Reasoning of Aggressive Person 

The aggressive person reasons that confronting persecutors is 
more reasonable than cooperating with them because 
cooperation shows weakness and invites being harassed and 
bullied. 

Confrontation, on the other hand, is thought to demonstrate 
bravery and forcefulness, which function to discourage 
supervisors from engaging in persecution. 
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So, the aggressive person reasons that confronting the persecutor is more 
reasonable than cooperating with them because cooperation shows weakness 
and invites being harassed and bullied. So, the basic idea is if you go along with 
the supervisor and conform to their wishes, then essentially what you’re doing is 
reinforcing the behavior and you are likely to be bullied more in the future. 
Confrontation, on the other hand, is thought to demonstrate bravery and 
forcefulness, which functions to discourage the supervisor from engaging in 
persecution. 
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To justify confrontation, aggressive individuals often seek 
evidence of hostile intent on the part of their supervisors. 

For example, they may attempt to build a case that their 
supervisors purposely overload them with work so that 
failing is inevitable. 


To justify confrontation, aggressive individuals often seek evidence of 
hostile intent on the part of their supervisors. This would be seeking out 
information to support their beliefs or one could think of it as a confirmatory bias. 
For example, they may attempt to build a case that their supervisors purposely 
overload them with work, so failing is inevitable. 
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Not only does such reasoning imply a strong bias to 
attribute behavior to malevolent purpose and harmful 
intent, but also it suggests an implicit (i.e., unconscious) 
propensity to assume that the powerful will inflict harm 
on the less powerful. 

This latent propensity maps into conscious reasoning by 
advancing an inference that one is the victim of inequity, 
exploitation, injustice, and oppression by those who are 
more powerful in one’s life (e.g., supervisors as well as 
parents, teachers, employing organizations, or institutions 
such as the Internal Revenue Service). 

10 


Not only does such reasoning imply a strong bias to attribute behavior to 
malevolent purpose and harmful intent, but also it suggests an implicit, that is to 
say unconscious, propensity to assume that the powerful will inflict harm on the 
less powerful. This latent propensity maps into conscious reasoning by 
advancing an inference that one is the victim of inequity, exploitation, injustice, 
and oppression by those who are more powerful than in one’s life. This might be 
supervisors as well as parents, teachers, employing organizations, or institutions 
such as the internal revenue service. 
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The thinking of aggressive individuals focuses 
progressively on how to obtain retribution and get even for 
the wounded pride, persecution, and disrespect that they 
believe they have suffered at the hands of their supervisors. 

Some form of aggression is judged to be reasonable and 
justified because it offers a means to restore respect and to 
exact restitution for the perceived wrongs. 


11 


Thinking of aggressive people focuses progressively on how to obtain 
retribution or get even for the wounded pride, persecution, and disrespect that 
they believe that they have suffered at the hands of their supervisors. Some form 
of aggression is judged to be reasonable and justified because it offers a means 
to restore respect into exact restitution for the perceived wrongs. 


55 




Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Volume 2, Part 2: Related Constructs 


Acts of retribution can take many forms, including 

• verbal aggression (e.g., threats, character assassination, 
intimidation, spreading of malicious gossip), 

• physical aggression (e.g., shoving, fighting, attacks with 
weapons), or 

• passive aggression (e.g., intentional nonattendance at 
team meetings, deliberate failure to return messages; 
habitual absenteeism/tardiness). 

12 


Acts of retribution can take many forms, including verbal aggression, 
physical aggression or passive aggression. 
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The framing and reasoning of our illustrative aggressive 
persons differs considerably from the reasoning that would 
be expected by prosocial individuals 


Aggressive Individuals 

• supervisors are persecutors 

• confrontation demonstrates power and 
discourages being victimized 

• cooperation and compromise 
demonstrate weakness and invite 
attack 

• retribution is justified 


Prosocial Individuals 

• supervisors are legitimate 
authority figures , perhaps even 
benefactors or protectors 

• confrontation serves as a catalyst 
for overt and continued conflict 

• cooperation and compromise 
defuse anger and promote long¬ 
term harmony 

• nothing has occurred to justify 
retribution 
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Now, let’s review just a little bit and add then the thinking of pro-social 
individuals. The framing and reasoning of our illustrative aggressive individuals 
differs considerably from the reasoning that would be expected from pro-social 
individuals. Whereas with aggressive individuals, supervisors are framed as 
persecutors, pro-social individuals would be much more prone to frame 
supervisors as legitimate authority figures perhaps even benefactors or 
protectors. While the aggressive person believes that confrontation demonstrates 
power and discourages being victimized, the pro-social person is likely to see 
confrontation as a catalyst for overt and continued conflict. Cooperation and 
compromise to the aggressive person demonstrates weakness and invites 
attack, whereas cooperation and compromise are seen by pro-social individuals 
as being used to defuse anger and promote long-term harmony. And the 
aggressive person believes that retribution is justified, whereas the pro-social 
person does not. Basically, aggressive individuals differ from pro-social 
individuals in terms of the adjectives they use to frame persons in authority, the 
behaviors of their supervisors that they select as being relevant indicators of their 
supervisor’s intentions, the assumptions they make about why supervisors 
behave as they do, and the arguments for and against aggression that they judge 
to be valid. Let me emphasize again that a large number of these differences are 
based on qualitative differences not just quantitative differences. 
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Why do aggressive people frame and reason as they do? 

The framing and reasoning of aggressive individuals is 
designed to justify a strong motive to harm others. 

Framing and reasoning are intended to enhance the rational 
appeal of engaging in behaviors that harm others. This is 
rationalization or justification. 

Aggressive people are generally unaware of this intent. 

They believe their reasoning is rational. 

Aggressive people can engage in this self-deception because 
the biasing of their reasoning (toward rationalizing 
aggression) occurs below the surface of their consciousness. 

14 


Why do aggressive people frame and reason as they do? Is the framing 
and reasoning of aggressive individuals designed to justify a strong motive to 
harm others? Framing and reasoning are intended to enhance the rational 
appeal of engaging in behaviors that harm others. This is rationalization or 
justification. Aggressive people are generally unaware of this intent; they believe 
that their reasoning is rational. Aggressive people can engage in this self- 
deception because the biasing of their reasoning, toward rationalizing 
aggression, occurs below the surface of their consciousness. I’m trained 
basically as a psychometrician, the use of terms like “unconscious” and “implicit” 
is tough, but we’re getting there. 
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Specifically, the reasoning of aggressive people is 
implicitly (unconsciously) shaped by biases whose function 
is to enhance the rational appeal of aggression. 

James (1998; James & Mazerolle, 2002) proposed the term 
“justification mechanisms” to identify these implicit biases. 


Specifically, the reasoning of aggressive people is implicitly shaped by 
biases whose function is to enhance the rational appeal of aggression. We 
propose that one could use the term “justification mechanisms” to describe these 
particular implicit biases. 
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Justification Mechanisms (JMs) 

(1) The Potency Bias involves an implicit proclivity to 
frame interactions with others as contests to establish 
dominance versus submissiveness. 


There are six justification mechanisms that we have identified in the 
literature that tend to characterize the reasoning of aggressive individuals. 
Potency Bias involves an implicit proclivity and can frame interactions with others 
as contests to establish dominance versus submissiveness. The basic idea is 
that people tend to perceive through a lens that judges behaviors of others in 
terms of its dominance versus submissiveness. Such framing is the cornerstone 
for justifying aggression as a demonstration of strength, bravery, or fearlessness, 
and not acting aggressively is logically attributed to weakness, fear, cowardice, 
or impudence. 
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JMs 

(2) The Hostile Attribution Bias consists of an implicit 
predilection to assume that malevolent purpose or harmful 
intent is the primary motivation underlying the behaviors 
of others (see Crick & Dodge, 1994). 


An aggressive person may thus rationalize aggression by reasoning that 
aggression is an act of strength or bravery that gains respect from others, or to 
show weaknesses to invite powerful others to take advantage of you. This is the 
Hostile Attribution Bias; it’s well known in the literature and consists of an implicit 
predilection to assume that malevolent purpose or harmful intent is the primary 
motivation underlying the behaviors of others. Basically, the actions of others 
pass through a perceptual prism primed to see hostile or malevolent intent. The 
attributions of hostile intent that follow are then employed by the aggressive 
person to rationalize his or her own hostile behaviors as acts of self defense 
intended to ward off physical or verbal attack. 
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JMs 

(3) The Retribution Bias involves an unconscious tendency 
to confer logical priority to reparation or retaliation over 
reconciliation. 


The Retribution Bias involves an unconscious tendency to confer logical 
priority to reparation or retaliation over reconciliation. This justification 
mechanism promotes reasoning that aggression is a justifiable response, if the 
intent of the response is to restore, respect or do exact restitution for a perceived 
wrong. Retaliation thus appears more reasonable than forgiveness, vindication 
appears more reasonable than cooperation, and obtaining revenge appears 
more reasonable than maintaining the relationship. This bias often underlies 
rationalizations for aggression engendered by wounded pride, challenged self¬ 
esteem, or perceived disrespect. 
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JMs 

(4) The Victimization by Powerful Others Bias is an implicit 
tendency to see everyday people (including oneself) as 
victims of inequity, exploitation, injustice, or oppression by 
powerful others. 


The Victimization by Powerful Others Bias is an implicit tendency to see 
everyday people, including oneself, as victims of inequity, exploitation, injustice, 
or oppression by powerful others. Powerful others include entities such as 
supervisors, teachers, and employment organizations. The framing of events 
hypothesis about cause and effect and confirmatory searches for evidence, both 
engender and reinforce unconscious inclinations to infer that people are being 
victimized by powerful others. This reasoning furnishes the foundation for 
rationalizing acts of aggression as warranted corrections of inequities or 
legitimate strikes against oppression. 
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JMs 

(5) Derogation of Target Bias consists of an unconscious 
tendency to characterize those one wishes to make (or has 
made) targets of aggression as evil, immoral, or 
untrustworthy. 


The Derogation of Target Bias, and this been the most difficult one for us 
to make work, consists of an unconscious tendency to characterize those one 
wishes to make, or has made, targets of aggression as evil, immoral or 
untrustworthy. And to infer or associate such negative traits with the target 
makes the target more deserving of aggression. 
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JMs 

(6) Social Discounting Bias is comprised by an implicit 
tendency to favor socially unorthodox and antisocial reasons 
as logically probable causes of social events and 
relationships. 


And finally, the Social Discounting Bias, and out of the group this 
analytically has shown to be the most important, is comprised by an implicit 
tendency to favor socially unorthodox and antisocial reasons as logically 
probable causes of social events and relationships. Reasoning shaped by this 
bias reflects disdain for traditional ideas and intentional beliefs. This reasoning 
will further evidence a lack of sensitivity, empathy, and concern for social 
customs, often accompanied by the absence of rational prohibitions against 
acting aggressively. Analyses of social events tend toward the cynical and critical 
with the proclivity to associate positive outcomes with aggression. 
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Measuring JMs 

How is it possible to determine whether JMs are 
instrumental in shaping a person’s reasoning? 

Why would we want to know this? 


Now, the basic idea of the measurement system is to measure these 
justification mechanisms. How is it possible to determine whether justification 
mechanisms are instrumental in shaping a person’s reasoning? And why would 
we want to know this? 
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Measuring JMs 

Reasoning that is shaped by JMs indicates a predisposition 
to behave aggressively. The person is prepared to 
rationalize aggression. 


Reasoning that is shaped by justification mechanisms indicates a 
predisposition to behave aggressively. This person is prepared to rationalize 
aggression and to engage in aggression. We are working basically from the 
assumption that only people who have aggressed in the past and intend to 
aggress in the future have needs of justification mechanisms for aggression. We 
can use the measurements of this, if we can obtain them, as a basis for 
diagnosing aggressive tendencies or as a foundation for scientific studies of 
implicit cognitive process. 
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Conditional Reasoning 

Illustrative problem for entry-level jobs: 

The old saying, "an eye for an eye," means that if someone 
hurts you, then you should hurt that person back. If you are 
hit, then you should hit back. If someone bums your house, 
then you should burn that person's house. 

Which of the following is the biggest problem with the "eye 
for an eye" plan? 

a. It tells people to "turn the other cheek." 

b. It offers no way to settle a conflict in a friendly manner. 

c. It can only be used at certain times of the year. 

d. People have to wait until they are attacked before they 
can strike. 

24 


This is a problem we’ve designed for entry level jobs for students. 
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Respondents believe that their critical intellectual skills 
guide their attempts to identify a logically correct 
conclusion to this problem. 

The demand appears intellectual because inductive 
reasoning is in fact required to solve the problem. 

However, accuracy of inductive analyses shaped by critical 
reasoning skills is NOT the primary issue here. 
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Respondents believe that their critical intellectual skills guide their 
attempts to identify a logically correct conclusion to the problem. The people are 
given the problems and they are told that it’s a reasoning test. Demand appears 
intellectual because inductive reasoning is required to solve the problem. 
However, accuracy of inductive analyses shaped by critical reasoning skills is not 
the primary issue here. 
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Our intent is that inductive seeking of the most 
reasonable answer be guided, unknowingly, by implicit 
assumptions about what constitutes rational behavior. 

We want reasoning to be shaped by either JMs for 
aggression or internalized prosocial values and 
ideologies. 

In a manner analogous to everyday reasoning about the 
rationality of aggression, what is judged to be the most 
reasonable answer to the problem is CONDITIONAL 
on whether analyses are shaped by JMs or prosocial 
proclivities. 
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Our intent is that inductive seeking is the most reasonable answer to be 
guided unknowingly by implicit assumptions about what constitutes rational 
behavior. We want reasoning to be shaped by either justification mechanisms for 
aggression or internalized pro-social values and ideologies. The idea here is that 
the pro-social values and ideologies are largely automated. In a manner of 
analogous everyday reasoning about the rationality of aggression, what is judged 
to be the most reasonable answer to a problem is conditional on whether 
analyses are shaped by justification mechanisms or pro-social proclivities. 
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d. People have to wait until they are attacked before they can 
strike. 

This alternative was designed to be logically attractive to 

aggressive individuals because it is grounded in their biases. 

Specifically, it 

• tacitly promotes retribution as being logically preferable 
to reconciliation (Retribution Bias) 

• is founded on the unstated assumption that the powerful 
will inflict harm on the less powerful unless the less 
powerful strike first (Victimization & Potency Biases) 
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For example, people have to wait until they are attacked before they can 
strike. This alternative was designed to be logically attractive to aggressive 
individuals because it’s grounded in their biases. Specifically, it tacitly promotes 
retribution as being logically preferable to reconciliation; we use the Retribution 
Bias to shape that one. And furthermore, is founded on the unstated assumption 
that the powerful will inflict harm on the less powerful unless the less powerful 
strikes first. And that is based on the Victimization and Potency Biases. 
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The logical credibility of this answer increases as JMs 
become more prominent in shaping reasoning. 

Selection of Alternative d as the most logically persuasive 
of the conclusions offered provides indirect evidence that 
the Retribution, Victimization, and Potency Biases are 
implicitly instrumental in shaping the reasoning of a 
respondent. 


Now, the logical credibility of this answer increases as justification 
mechanisms become more prominent in shaping reasoning. And this is another 
way of looking at conditional reasoning, which is that the logical credibility of an 
answer is conditional on the prominence of justification mechanisms in shaping 
reasoning. Selection of alternative “d” as the most logically persuasive of the 
conclusions offered provides indirect evidence that the Retribution, Victimization 
and Potency Biases are implicitly instrumental in shaping the reasoning of a 
respondent. 
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b. It offers no way to settle a conflict in a friendly manner. 

This alternative was designed to be logically attractive to 
prosocial individuals because it 

• promotes a prosocial counterbalance to the antagonistic 
and provocative tenor of the aggression alternative, and 

• is grounded in the unstated assumption that conflict is 
logically less reasonable than compromise and 
cooperation. 
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Let’s go to the second alternative. The eye-for-an-eye offers no way to 
settle a conflict in a friendly manner, this alternative was designed to be logically 
attractive to pro-social individuals because it promotes a pro-social 
counterbalance to the antagonistic and provocative tenor of the aggression 
alternative. Furthermore, it is grounded in the unstated assumption that conflict is 
logically less reasonable than compromise and cooperation. 
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The logical plausibility of this answer increases as 
prosocial factors become more influential in shaping 
reasoning. 

Selection of Alternative b provides indirect evidence that 
the Retribution, Victimization, and Potency Biases are 
NOT instrumental in shaping reasoning. 


The logical plausibility of this answer increases as pro-social factors 
become more influential in shaping reasoning. So, selection of this alternative is 
indirect evidence that the Retribution, Victimization, and Potency biases are not 
instrumental in shaping the reasoning of response. 
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a. It tells people to "turn the other cheek." 

c. It can only be used at certain times of the year. 

These two alternatives are included to enhance the face 
validity of the task and to protect the indirect nature of 
measurement. 

Alternatives a and c are meant to be clearly illogical and 
rejected by respondents (which is usually the case). 
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Now, the last two alternatives are included to enhance the face validity of 
the task and to protect the indirect nature of measurement. These alternatives 
are meant to be clearly illogical and rejected by respondents, which is usually the 
case; very few people ever pick these distracters. I debated whether to show you 
some of our new items. We find that respondents have no troubles with this, but 
psychologists do once in a while. So, what I decided to do primarily to enhance 
the sophistication of these problems is to take these distracters and make them 
real distracters. I spent a few months early in the game doing a content analysis 
of inductive reasoning problems and it became pretty clear that the trick to writing 
a really good inductive reasoning problem, well we’ll see if people here will 
disagree with me, is writing good distracters and writing good distracters is tough 
business. So, what we’ve done is we’ve crossed justification mechanisms with 
intelligence. Basically, we still have two distracters - they are both wrong, but one 
of them is logically wrong and a measure of a justification mechanism and one of 
them is logically wrong and a measure of pro-social values. So it’s a fully crossed 
model and there are much more sophisticated items. I don’t have much data on 
them yet, and I have no validity so I am going to wait until I present those but we 
are working on some more sophisticated techniques. 
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How this illustrative problem is solved is dependent, or 
CONDITIONAL, on the relative degrees to which JMs for 
aggression versus prosocial values and ideologies are 
instrumental in shaping reasoning. 

The problem is referred to as a CONDITIONAL 
REASONING PROBLEM. 

This psychometric approach is referred to as the 
“conditional reasoning measurement system” (James, 

1998; James & Mazarolle, 2002) 
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How this illustrative problem is solved is dependent or conditional on the 
relative degrees to which justification mechanisms for aggression versus pro¬ 
social values and ideologies are instrumental in shaping reasoning. The problem 
is referred to as a conditional reasoning problem. The psychometric approach is 
referred to as the conditional reasoning measurement system. 
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Animals in the wild often fight to determine who will breed. 
The reason for fighting, sometimes to the death, is that only 
the strongest will be allowed to breed. Fighting is designed 
to insure that only the most powerful and fit animals survive 
and reproduce. 

Humans usually reproduce without first determining who is 
the strongest. This suggests that: 

a. physical strength is usually not important in determining 
who will contribute to society. 

b. animals breed most often in the Fall. 

c. the study of genetics is becoming less popular. 

d. the human race, on the average, is becoming physically 
weaker. 


I have a couple more problems and I’ll let you play with these for a 
second. Which one is the aggression response? D, right. And the pro-social 
response? A, right. I’ve learned the hard way not to ask people how they got to 
this because they might tell you. 
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JMs 

1. Potency Bias: a propensity to frame interactions with others through a prism 
of dominance versus submissiveness (or strength versus weakness). 

2. Hostile Attribution Bias: a tendency to seek out malevolent purpose or 
harmful intent as the motivations for the actions of others. 

3. Retribution Bias: a predilection to favor vengeance, retribution, and 
retaliation over reconciliation, cooperation, or compromise. 

4. Derogation of Target Bias: a predilection to characterize the target of 
aggression as evil, immoral, untrustworthy, or corrupt, and thus more 
deserving of being assailed. 

5. Victimization by Powerful Others Bias: tendency to frame self as a victim 
and to see self as being exploited and taken advantage of by the powerful 
(e.g., employing organization). 

6. Social Discounting Bias: tendency to call on socially unorthodox and 
frequently antisocial beliefs to interpret and to analyze social events and 
relationships. 

34 


So, which justification mechanism bias for the aggression response? 
Potency, right. 


78 




Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Volume 2, Part 2: Related Constructs 


One-half of marriages end in divorce. One reason for the 
large number of divorces is that it is easy and quick to get a 
divorce. If a couple can agree on how to split their property 
fairly, then they can get a divorce simply by filling out forms 
and taking them to court. They do not need lawyers. 

This means that: 

a. people are older when they get married. 

b. if one's husband or wife hires a lawyer, then he or she is 
not planning to play fair. 

c. couples might get back together if getting a divorce took 
longer. 

d. more men than women get divorced. 35 


One more. The aggression response? D, right. And the pro-social 
response? C, right. 
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JMs 

1. Potency Bias: a propensity to frame interactions with others through a prism 
of dominance versus submissiveness (or strength versus weakness). 

2. Hostile Attribution Bias: a tendency to seek out malevolent purpose or 
harmful intent as the motivations for the actions of others. 

3. Retribution Bias: a predilection to favor vengeance, retribution, and 
retaliation over reconciliation, cooperation, or compromise. 

4. Derogation of Target Bias: a predilection to characterize the target of 
aggression as evil, immoral, untrustworthy, or corrupt, and thus more 
deserving of being assailed. 

5. Victimization by Powerful Others Bias: tendency to frame self as a victim 
and to see self as being exploited and taken advantage of by the powerful 
(e.g., employing organization). 

6. Social Discounting Bias: tendency to call on socially unorthodox and 
frequently antisocial beliefs to interpret and to analyze social events and 
relationships. 

36 


And the justification mechanism? 
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Conditional Reasoning Test for Aggression (CRT-A) 

Composition: 22 conditional reasoning (CR) problems and three inductive 
reasoning problems. 

Scoring: "+1" for aggression (AG) alternative; “0” for every logically incorrect 
alternative; and ”-T' for prosocial or nonaggressive (NA) alternative. Scores are 
summed to furnish a composite score on the “Justification of Aggression Scale” or 
“JAGS.” 

Interpretation of scores on the JAGS: 

High score: JMs for aggression are instrumental in guiding and shaping 
reasoning. Respondents are “justifiers,” which connotes that they are implicitly 
prepared and willing to engage in some form of aggressive behavior. 

Low score: JMs are not instrumental in guiding and shaping reasoning. 
Respondents are not implicitly prepared and willing to engage in aggressive 
behavior. 

Mid-range score: JMs are only sporadically instrumental in shaping reasoning. 
The implicit readiness to aggress is therefore likely to be only modest. 

37 


The instrument itself is composed of 22 problems, including the three you 
have just seen, and we have used a number of different scoring methods. It 
changes, but one point for an aggression response, zero for logically incorrect 
alternative, minus one for pro-social, this gives us the cleanest factor structure, 
the most straightforward way to simply score the number of aggression 
responses or number of pro-social depends on which way you want to orient the 
scale. We refer to this as the Justification of Aggression Scale or JAGS. The 
interpretation of the scores is that a high score reveals that justification 
mechanisms for aggression are instrumental in guiding and shaping and 
reasoning. Respondents are justifiers, which connotes that they are implicitly 
prepared and willing to engage in some form of aggressive behavior. A low score 
reveals that justification mechanisms are not instrumental in guiding and shaping 
and reasoning. Respondents are not implicitly prepared or willing to engage in 
aggressive behavior. 
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Verbal /Visual Conditional Reasoning Test (VCRT). 

For less adept readers, a test based on a “verbal-visual” version of a 
subset of the CR problems was designed to have a threshold reading 
level of approximately the fifth to sixth grade (Green & James, 

1999 ). 

Referred to as the “VCRT,” this test consists of bare-bones versions of 
CR problems. The problems are presented both verbally and in written 
form using a videocassette player and television. 

The written component consists of simplified prose, which is overlaid 
on a photograph consistent with the basic theme of the CR problem. 

The current VCRT contains 14 CR problems, 12 of which are shared 
with the CRT-A. Work continues on converting CRT-A problems to 
the VCRT format. Scoring and interpretation of scores on the VCRT 
are analogous to procedures employed on the JAGS. 


We did this for an unnamed Southeastern University Football team; it’s a 
Verbal/visual Conditional Reasoning Test, VCRT. Basically, what we do is 
simplify the problems or take it to three alternatives: an aggression response, a 
pro-social response, and one distracter and then overlay it on a photograph that 
was searched through the files of Life Magazine to find photographs that 
appeared to best illustrate this particular problem. It is also is given verbally; the 
problem is written in a simplified version and the problem is given verbally. So far 
we have 14 problems in this instrument, 12 of which are shared with the 
Conditional Reasoning Test for aggression. 
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Empirical Evaluations 

Distributions on CR problems and JAGS were skewed. 
Approximately 12% of respondents considered moderate 
to strong justifiers. 

CR problems factored by the JMs. 


Empirical evidence arises from the fact that distributions on the CR 
problems and JAGS were skewed. Approximately 12 percent of respondents are 
considered moderate to strong justifiers or moderately strong aggressive. 
Additionally, the factor analyses demonstrate that the problems do indeed factor 
by justification mechanism, although there is a fair degree of complexity; eight 
particular items did not necessarily pick up just one justification mechanism. 
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Reliabilities 


Factorial (n = 

1603) 


Factor 1 

.87 


Factor 2 

.82 


Factor 3 

.81 


Factor 4 

.76 


Factor 5 

.74 


Internal Consistency 


CRT 

.76 (n= 1603) 


VCRT 

.78 (n = 225) 


Alternative Form 


CRT/VCRT 

.82 (n = 276) 

40 


If you look at the factorial reliabilities, they tend to run fairly high. The 
internal consistency estimates tend to run in the mid seventies, sometimes the 
upper seventies. I plan to build a sufficient number of items that we have fairly 
internally consistent measures of each justification mechanism, but we are not 
there yet, we don’t have enough problems. With an alternative form, where we 
gave the conditional reasoning test right at the first of the semester to a group of 
students who then took the VCRT toward the end of the semester, we calculated 
the correlation between them, which was .82, which demonstrates that the two 
ways of measuring are pretty similar as well as the stability, so that is pretty 
good. 
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Uncorrected Validities for Scores on JAGS 


Sample # & Criterion 

Sample 

Instrument 

Design 

Uncorrected 





Validity* 

1. Performance Ratings 

140 Patrol Officers 

CRT 

Concurrent 

-.49 

2. Absences from Class 

188 Undergrads 

CRT 

Predictive 

.37 

3. Lack of Truthfulness 

60 Undergrads 

VCRT 

Experiment 

.49 

about Extra Credit 





4. Absences from Work 

97 Nuclear Facility Operators 

CRT 

Postdictive 

.42 

5. Conduct Violations 

225 Undergrads 

VCRT 

Postdictive 

.55 

6. Attrition 

135 Restaurant Employees 

CRT 

Predictive 

.32 

7. Absences from Work 

105 Package Handlers 

CRT-A 

Predictive 

.34 

8. Work Unreliability 

111 Temporary Employees 

CRT-A 

Predictive 

.43 

9. Theft 

95 Undergrads 

CRT-A 

Experiment 

.64 

10. Hard Fouls & Fights 

191 Undergrads 

CRT-A 

Predictive 

.38 

in Intramural Basketball 





*A11 correlations 

are statistically significant (p<.05). 

41 


We conducted a number of validity studies, most with real people in real 
world situations. For example, patrol officers, nuclear facility operators, 
restaurant employees, package handlers, temporary employees, etc. We’ve also 
conducted a couple of experiments. In study three, we tested to see if the 
students would be truthful about the amount of extra credit they deserved in a 
particular exercise. In another study that was done by Terri Mitchell from the 
University of Washington, is whether students would steal a gift after they had 
been clearly frustrated in an exercise. What we found was that a number of 
people had questions about whether these criteria are actually measures of 
aggression. And so in response to that, we did a review of the literature and 
found basically aggression is an integral component of organizational retribution, 
counter-productive performance, dysfunctional resistance tactics, obstructive 
behaviors and workplace deviants. You can see where the research has been 
going the last ten years in organizational psychology. This is because many, if 
not most deviant retaliatory, counter productive, resistant and destructive and 
displaced behaviors, involve hostile attempts to harm an organization or its 
constituents by exacting retribution, revenge or retaliation in ways that disrupt 
work schedules, impede productivity, weaken morale, undermine authority, 
encourage rebelliousness and get even with the boss or co-workers. 
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Uncorrected Validities for Scores on JAGS 


Sample # & Criterion 

Sample 

Instrument 

Design 

Uncorrected 





Validity* 

1. Performance Ratings 

140 Patrol Officers 

CRT 

Concurrent 

-.49 

2. Absences from Class 

188 Undergrads 

CRT 

Predictive 

.37 

3. Lack of Truthfulness 

60 Undergrads 

VCRT 

Experiment 

.49 

about Extra Credit 





4. Absences from Work 

97 Nuclear Facility Operators 

CRT 

Postdictive 

.42 

5. Conduct Violations 

225 Undergrads 

VCRT 

Postdictive 

.55 

6. Attrition 

135 Restaurant Employees 

CRT 

Predictive 

.32 

7. Absences from Work 

105 Package Handlers 

CRT-A 

Predictive 

.34 

8. Work Unreliability 

111 Temporary Employees 

CRT-A 

Predictive 

.43 

9. Theft 

95 Undergrads 

CRT-A 

Experiment 

.64 

10. Hard Fouls & Fights 

191 Undergrads 

CRT-A 

Predictive 

.38 

in Intramural Basketball 





*A11 correlations 

are statistically significant (p<.05). 

41 


To avoid punishment, these processes seldom involve outright violence or 
acts that are easily detectible as aggression. Rather, they focus on indirect 
passive-aggressive behaviors such as failing to come to work, or coming to work 
late, stealing from those seen as guilty of injustices, lying to authority figures to 
regain face and obtain retribution for being disrespectful in performing in poor 
unreliable or improper matters. In short, we got a little defensive about this and 
said look, a lot of these have aggressive types of connotations or can be 
products of aggressive behavior, doesn’t mean that all of them are, but the 
probability is pretty good. After all that we went ahead and did an aggression 
study. So, we studied intramural basketball at a large Southeastern University, 
and we looked at a lot of things. For example, we looked at hard fouls and fights 
in intramural basketball, and basically a hard foul is where someone is hit and 
often times goes to the ground or a fight. We had people watch every one of the 
games and recorded this and the students before the start of the season, took 
the conditional reasoning test. 


86 




Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Volume 2, Part 2: Related Constructs 


Uncorrected Validities for Scores on JAGS 


Sample # & Criterion 

Sample 

Instrument 

Design 

Uncorrected 





Validity* 

1. Performance Ratings 

140 Patrol Officers 

CRT 

Concurrent 

-.49 

2. Absences from Class 

188 Undergrads 

CRT 

Predictive 

.37 

3. Lack of Truthfulness 

60 Undergrads 

VCRT 

Experiment 

.49 

about Extra Credit 





4. Absences from Work 

97 Nuclear Facility Operators 

CRT 

Postdictive 

.42 

5. Conduct Violations 

225 Undergrads 

VCRT 

Postdictive 

.55 

6. Attrition 

135 Restaurant Employees 

CRT 

Predictive 

.32 

7. Absences from Work 

105 Package Handlers 

CRT-A 

Predictive 

.34 

8. Work Unreliability 

111 Temporary Employees 

CRT-A 

Predictive 

.43 

9. Theft 

95 Undergrads 

CRT-A 

Experiment 

.64 

10. Hard Fouls & Fights 

191 Undergrads 

CRT-A 

Predictive 

.38 

in Intramural Basketball 





*A11 correlations 

are statistically significant (p<.05). 

41 


You take a look at the validities in this table you will see that the average 
validity runs about .44. Putting that in a context, you take a look at somewhere 
around a dozen or so meta-analyses that have been done in the last 10-12 years 
on validity and personality in industrial situations, almost inevitably based on self- 
report, the average validity is .12. So, you are looking at the difference and these 
are uncorrected validities, so you’re looking at a difference of an average of .44 
against .12 and I think if anything that makes a pretty good case that implicit 
personality has a place in psychology. 
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Correlations between Scores on the JAGS 
and Critical Intellectual Skills 


Study 

Intelligence Scale 

n 

Sample 

Correlation 

2 

ACT 

188 

Undergrads 

-.06 

3 

ACT 

60 

Undergrads 

-.05 

5 

ACT 

225 

Undergrads 

-.08 


*p < .05 

Note . ACT = American College Testing. 


There is no correlation between the JAGS and critical intellectual skills. If I 
were picking the test this school used, I would have used the SAT, but we have 
the ACT here, so sorry about that. Basically, how smart you are doesn’t 
determine how you respond to the questions. 
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Relationships between Scores on the JAGS and Gender 

Sample 

Composition 

% Female 

Point Biserial 

Biserial 




Correlation 

Correlation 

2 

188 undergrads 

.34 

-.08 

-.10 

3 

60 Undergrads 

.60 

-.22* 

-.29* 

5 

225 Undergrads 

.49 

-.20* 

-.25* 

6 

120 Restaurant Employees 

.66 

-.06 

-.08 

7 

105 Package Handlers 

.26 

.04 

.05 

8 

111 Temporary Employees 

.36 

.07 

.09 

9 

95 Undergrads 

.55 

.00 

.00 

Male 

= 0, Female = 1. 




*p < 

.05 



43 


In terms of relationships between the scores on the Conditional Reasoning 
Test and gender, in a few cases we find that males tend to score a little more 
highly than females but that’s certainly not consistent. Our basic position on this 
is that males and females have justification mechanisms and that the primary 
difference in manifestations of aggression comes in overtly; how it’s manifested 
is not necessarily in the implicit possession of rationalization mechanisms. 
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Relationships between Scores on the JAGS and Race 


Sample 

Composition 

% 

Race 

Relationship 

2 

188 undergrads 

.90 

White (0) 

r = .06 1 



.10 

African American (1) 

(r = .10) 2 

7 

105 Package Handlers 

.23 

White 

F = 1.43 



.52 

African American 




.24 

Hispanic 




.01 

Asian 


8 

111 Temporary Employees 

.82 

White (0) 

r=.07 1 



.18 

African American (1) 

(r=.10) 2 

9 

95 Undergrads 

.57 

White 

F = .16 



.38 

Asian 




.04 

Hispanic 




.01 

African American 



*p < .05 

'Point Biserial Correlation 
2 Biserial Correlation 


There is no correlation between the JAGS and race. 
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Correlations between Scores on the JAGS 
and Self-Report Measures of Aggression 


Sample 

Composition 

Alternative Measure 

Correlation 

3 

60 Undergrads 

PRF Aggression 

.14 



PRF Dominance 

.05 



PRF Impulsivity 

.11 

5 

225 Undergrads 

NEO-PI-R Angry Hostility 

.26* 



NEO-PI-R Dutifulness 

-.18* 

9 

95 Undergrads 

Aggression Questionnaire 

.24* 

10 

191 Undergrads 

NEO-PI-R Angry Hostility 

.002 


*p <. 05 

PRF = Personality Research Form (Jackson, 1967). NEO-PI-R = 
NEO Personality Inventory (Revised) (Costa & McCrae, 1985, 
1992). Aggression Questionnaire (Buss & Perry, 1992). 45 


This is the one that always gets me in trouble, which is the correlation 
between the scores on the JAGS and self-report measures of aggression. For 
the most part, the correlation is either non-significant or significantly low. When I 
teach a personality class and when we get to this point right here and the 
industrial psychologists look at this and they say “Aha, lack of convergent validity, 
you’re in trouble with construct validity, your implicit correlates with your explicit 
and you have problems.” Then the clinical psychologists look at this and say, 
“What else is new? There obviously is a conflict between implicit and explicit, 
why would you expect there to be a correlation?” And what’s interesting is that 
from that day forward, the clinical psychologists sit on one side of the room and 
the industrial psychologists sit on the other side. We are conducting a number of 
studies now, looking at the interaction between implicit and explicit using 
integrative models that McClellan proposed. We are finding some fairly good 
success with those. Basically, what you do is cross self-reports of aggression 
with the conditional reasoning measure of aggression and then you look for 
consistencies and inconsistencies. And what we’re finding is that the most 
aggressive people are people who on the self-report measure say that they’re 
aggressive and with our measures we say yes, you have the rationalization to 
back them up. But, we are finding also that some of the least aggressive with 
overt measures are people who say they’re not aggressive but have the 
justification mechanisms in place. Unless we go to a passive-aggressive 
measure, then things change. We intend to get there but we are not there yet. 
This is a fascinating area and we will keep with it. 
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Multiple Regression-Dominance Analyses 


Sample 

Criterion 

Multiple R 

Variable 

Zero-Order r 

Beta 

Relative 






Weight 

Importance 

3 

Lack of 

.55* 

JAGS 

.49* 

.51* 

82.83% 


Truthfulness about 


ACT 

-.07 

-.12 

2.89% 


Extra Credit 


PRF Aggression 

.16 

.17 

8.11% 




PRF Impulsiveness 

.14 

.14 

6.05% 




PRF Dominance 

.05 

.04 

.72% 

5 

Student Conduct 

.61* 

JAGS 

.55* 

.53* 

77.60% 


Violations 


NEO Angry Hostility 

.26* 

.21* 

16.00% 




NEO Dutifulness 

-.18* 

-.13* 

6.43% 

9 

Theft 

.70* 

JAGS 

.64* 

.57* 

71.7% 




Aggression Quest. 

.44* 

.30* 

28.3% 

10 

Hard Fouls & 

.45* 

JAGS 

.38* 

.38* 

74% 


Fights 


NEO Angry Hostility 

.22* 

.22* 

26% 


*p <. 05 


We are back at the old question, implicit versus explicit. I don’t really like 
to frame it this way, and sometime as soon as I get some slides prepared on 
these channeling models, I’m going to get rid of this one. But, the basic idea was 
if we put these things into straight multiple regressions, now these don’t have the 
interaction terms in them, and we compute a dominance analysis which takes a 
look at the relative importance of each of the predictors to the multiple R 
squared, you will see that the JAGS or the Conditional Reasoning measure wins 
out every time. I consider this important because for years as I did research on 
organizational and psychological climate, I became convinced that the most 
salient feature in terms of perceptions of organizational situations and 
organizational environment is who is doing the proceedings. Then, to relate that, 
it doesn’t mean that the organization is not important, it just means that there is a 
strong interaction with who is doing the proceeding. Then the next step was then 
to relate the climate perceptions to personality factors to find out what it was 
about people that were leading to differences in perceptions of the environment. 
These were fairly complex models with person-environment fit and the further we 
got into this the more convinced I became that maybe the hypotheses weren’t so 
bad, but it was the measurement system and maybe on both sides of the 
equation. The self- report measures weren’t sophisticated enough to pick up 
some of the hypotheses we had and that is what got me into this venture. 
Basically, you can’t walk away from meta-analyses that say that personality 
counts for one-percent of the variance in people’s behavior and in organizations; 
it’s much more important than that. 
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Closing Comments 

The JAGS predicts a reasonably wide array of behavioral criteria, 
many of which have been salient to civilian, government, and 
military organizations for some time (e.g., attrition). 

The average validity of .44 compares favorably with validities 
produced by measures of critical intellectual skills, and surpasses the 
average, uncorrected validity of .12 produced by self-report 
personality inventories. 

Due to their indirect nature, it is reasonable to expect that CR 
problems are resilient to faking. Initial findings are that the CRTs for 
aggression and achievement motivation are resistant to socially 
desirable responding and impression management when administered 
under normal circumstances. 

47 


The JAGS predicts a reasonably wide and array of behavioral criteria, 
many of which have been salient to civilian, government, and military 
organizations for some time. Due to their indirect nature, it is reasonable to 
expect that Conditional Reasoning problems are resilient to faking. Initial findings 
are that the CRTs for aggression and achievement motivation are resistant to 
socially desirable responding and impression management when administered 
under normal circumstances. 
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DISCUSSION 


DISCUSSION 

DR. JAMES: Thank you. 

DR. KYLLONEN: We have some time for questions. 

DR. MAYER: Is there ever a time when these justification mechanisms 
might need to be adjusted? 

DR. JAMES: Well, I am not going to make a moral judgment, but yeah, 
sure. Basically it is one rationalizing aggression. The way I prefer to look at this, 
is say that you were a member of the French resistance in Paris, and you 
engaged in aggression like sabotage, is that aggression? One can get into 
debates on that sort of thing, but the idea is that with aggressive people, they 
take that type of situation, which you have a justifiable form of aggression and try 
to apply it to situations in which it is not justified. That is the key to what we are 
trying to look at. Would there be situations in which the aggression is justified, 
therefore the rationalization, in which case it might not even be a rationalization? 
Sure! 


DR. GRANDEY: Can we teach people who have problems like feeling, 
and wrong values about these things and have them and change what they 
believe? 

DR. JAMES: I’m in the measurement business; I’m not in the change 
business. I suspect you could. 
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DISCUSSION 


PARTICIPANT: Would you comment on the relationship of what you’ve 
been assessing with psychopathy? 

DR. JAMES: Well Bob Hogan tells me it’s pretty high, so I would suspect 
with the implicit paranoia that goes through a lot of those things; we’ve haven’t 
studied it, but I assume it’s high. 

DR. IRVINE: The only reason I’m asking this is because as a 
psychometrician I found myself in a large cognitive department for many years. 
I’ve been doing what you’ve been doing; all of my cognitive friends have said 
“Have you read the materials on bias and reasoning because we publish most of 
it?” So my question follows. You have analyses that show that you have factorial 
strength in what you are testing. Have you thought of getting some of the other 
items or other kinds of bias and reasoning and putting them up against yours? 

DR. JAMES: Yes. 

DR. IRVINE: And have you done it? 

DR. JAMES: No, these are all are opportunities. Another question has 
been, “Have you looked at it with projective techniques?” We’ve had one study 
where we’ve been able to do that. Right now, frankly this is not a magnet for 
funding. We just got a four million dollar grant to study climate, from NIMH. I 
haven’t had an original idea in climate in 20 years. But it’s a proven commodity 
so we can get a lot of money to do research. The Navy gave us a $100,000 
grant, six or seven years ago, and that’s about the only funding we’ve had for the 
conditional reasoning research. 
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DISCUSSION 


DR. STRICKER: I wonder if the difference between the validity of your 
measure and the self- report measure in the meta-analyses is actually less than 
we might think. In the meta-analyses they are pooling across all kinds of different 
criteria and all kinds of variables within these different kinds of criteria, whereas 
you are focusing on a particular kind of variable. 

DR. JAMES: Yes, it’s not the fairest of comparisons. Personality ran into a 
lot of trouble in industrial psychology 25-30 years ago and in effect, they publicly 
called a moratorium on personality research. And the question one has to ask 
right now is, “Is personality in any better shape now than it was 25 or 30 years 
ago?” Well, there are probably some better measures and there’s probably some 
better theory, but the validities are about the same. Before we get into another 
problem where we have to justify doing research on personality, which is where 
these meta-analyses are taking us, I think someone has to stand up and say 
something. 

DR. MATTHEWS: The person actually asked the question I was going to 
ask, so to go back to the personality questionnaires and their validity. What the 
meta-analyses show, is if you take confirmatory studies or theory-driven studies, 
those give you validities of .4 or so (a paper by Joyce Hogan in JAP this year) 
which is about the same level as the implicit measure. So, the advance has not 
so much been in the personality questionnaires, but in getting better theory, 
which allows you to predict the criteria which the scale is going to predict. 
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DR. JAMES: I’m afraid I will have to disagree with you on that. I was very 
specific about saying uncorrected validities, and the validities that you are talking 
about are corrected validities. I have had a rather long history of debates with 
people concerning validity generalization and I particularly don’t like the idea of 
correcting validities with an assumed reliability of a criterion of .52, which is what 
pumps those validities up. So, if you put those on a more even keel, either 
correct both, I would frankly be embarrassed to correct those, besides I won’t use 
a .52 reliability. So if you look at the uncorrected and this differential, I looked at 
the Hogan and I think that is a little higher, but the uncorrected validity is still not 
that high. So you have to compare uncorrected to uncorrected or corrected to 
corrected, in that case the differential is there. Besides we also have the 
empirical data with four studies now, where consistently we find that difference in 
validity with the dominance studies. 

DR. KYLLONEN: One more quick question ... anybody? 

DR. LEN WHITE: I wanted to ask and I think this is going to require you to 
speculate. Suppose you had instead choices of aggressive or non aggressive 
alternatives, based on the same justification mechanisms that varied moderately 
despite the specification, would you expect more of aggressive people, for them 
to be more able to more differentiate those levels of specification, than the less 
aggressive people? 
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DISCUSSION 


DR. JAMES: Well, I do have some p-values on that new instrument and 
I’m framing that now in terms of the question you’ve asked. Off the of top of my 
head, no and when you look at the cross model in terms of proportions, no. But, I 
could be wrong on that, I would have to look at the data to answer the question. 
The discussion that we had yesterday is a real contribution and that was the idea 
that we could look at each of the alternatives, on a Likert scale and measure the 
probability that that is logical. What I like about that is it gives us the opportunity 
to add more alternatives to each scale, to enhance the measurement system. But 
no, I don’t see those differences in specifications, but then again.... 

PARTICIPANT: It may make a difference? 

DR. JAMES: Yeah, but it’s set up so it’s a qualitative difference, so the 
cognitive complexity would really be on one side of the aggressive people looking 
at various subtleties. But on the pro-social side, you have the same thing, all they 
have to do is reject, and it’s set up to do that, then you could build those 
subtleties in on both sides. 
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Isms and Other Personality Constructs 

Gerard Saucier 
University of Oregon 

Conference on "Emotional Intelligence: 
Knowns and Unknowns" ETS, Princeton NJ 

14 November 2003 


WHAT IS BEYOND THE BIG FIVE? ISMS AND OTHER PERSONALITY 
CONSTRUCTS 

DR. KYLLONEN: I want to introduce our next speaker, who is going to be 
taking us in a slightly different direction. By the way the theme of this particular 
session is things that are related constructs to emotional intelligence so we are 
touching on personality, and things that are related to personality. Next speaker 
is Gerard Saucier from the University of Oregon. 

DR. SAUCIER: I’m aware that some of you may have been sitting in the 
same seat for two hours now, so if you want to take a 7 th inning stretch, there 
won’t be any patriotic songs. Pat said this was going in a different direction than 
the previous presentation; that is not entirely true, I noticed that the previous 
presentation involved beliefs and this particular presentation is also going to go in 
the direction of pointing the way in which peoples beliefs whether implicit or 
explicit can supplement personality and more traditional self report personality 
measures. I have become aware that an issue with regard to emotional 
intelligence is overlap with personality and that one critique that has been made 
particularly of self-report measures of emotional intelligence is that they may 
overlap with the Big 5, be highly predicted with the Big 5. 
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The Study of Individual 
Differences 

A Preliminary Basic Question: 

Which individual differences are most worth 
measuring? 

•In personality 
•In abilities 

* And what about social attitudes, or 
ideological beliefs? 


What I want to do today is focus defining what the content of personality 
measures is as an area of individual differences and what sorts of aspects of 
personality or other important individual differences are not captured on current 
measures current popular measures. I must warn you that this is going to lead 
me into talking about people’s attitudes about politics, religion and money and 
these are somewhat taboo topics and they have been that way in fact for 
personality psychologists but I think this presentation will indicate that we can 
avoid those topics somewhat at our peril. 

For people in many research areas, attitudes of that nature are something 
of a nuisance, you would rather keep completely clear of that. But even if that is 
your attitude, it is useful to know how to measure them so you can control for 
them and make sure you have got some accurate measures. There are other 
aspects that psychologists should be interested in such as international affairs, 
violent social movements, relationship compatibility, where these kinds of 
contracts and measures may really be more predictive and useful than self report 
personality measures. 
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The Study of Individual 
Differences 

A Preliminary Basic Question: 

Which individual differences are most worth 
measuring? 

•In personality 
•In abilities 

* And what about social attitudes, or 
ideological beliefs? 


In the study of individual differences an important basic question is which 
individual differences are most worth measuring. I got into the study of individual 
differences by taking a class with Lou Goldberg about 15 years ago, and I recall 
the first day in class he started talking about something and this is exactly the 
question that I asked him, and it’s been sort of prescient to what my research 
career has been about. It is an important question with regard to personality with 
regard to abilities and it’s been asked a lot with regards to those areas. But I’m 
also going to get today into what individual differences are most important in 
regards to social attitudes or ideology beliefs. Now in determining which are the 
most important variables to measure, one recourse might be to authority which is 
to say you respect somebody as an expert so you believe what they about what 
the most important variables are. 


102 




Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Volume 2, Part 2: Related Constructs 


The Lexical Rationale for Variable 
Selection: 

* The most important human attributes are 
encoded in language. 

* The degree of representation of an attribute 

in language has some correspondence with 
the general real-world social importance of 
the attribute 

AND...the most important attributes, or factors 
made up of attributes, should demonstrate 
cross-cultural generalizability 


One alternative to this, is to study the language, because of the rationale 
could be that the most important human attribute tends to become encoded in 
language and the degree of representation of an attribute of language has some 
correspondence with it’s social importance, therefore you could learn something 
about which is the most important variables to study by looking at what is 
emphasized in a language. This is a necessary by not sufficient criteria, it’s not 
saying that all the important variables are a language, or you should limit yourself 
to what’s in a language, it’s more a matter of whatever is emphasized in 
language you know should be included in your model somewhere, supplemented 
perhaps probably by other things. 

This is basically what’s been called the lexical hypothesis, although I don’t 
really think of it as a hypothesis, so much as a rationale for variables selection. 

An additional criterion would be or part of that hypothesis evolved is that the most 
important attributes or factors that you derive from many attributes should 
demonstrate some kind of a cross cultural generalizability. 
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The Lexical Rationale for Variable 
Selection: 

* The most important human attributes are 
encoded in language. 

* The degree of representation of an attribute 

in language has some correspondence with 
the general real-world social importance of 
the attribute 

AND...the most important attributes, or factors 
made up of attributes, should demonstrate 
cross-cultural generalizability 


Now cross cultural generalizability, I consider an underused criterion and a 
lot of areas of psychology, we use a sort of coca cola model, which is to say that 
we develop our product in a western country and then export it world wide, with 
the assumption, what’s valid for college students in Indiana is valid for 
everybody. The kind of approach I’m going to describe with regards to lexical 
studies of personality actually enables us to use a different approach, where we 
might be able to develop a model that works simultaneously in a wide variety of 
cultural settings. And we may be able to locate a model that’s not primarily from 
one cultural area or one group of languages but rather it either is universal or 
fairly well distributed across global languages. 

There have been by my count 14 lexical studies in different languages, 
these are studies where the personality terms typically adjectives are collected 
from a dictionary and they’re reduced typically the most frequently used terms 
and then these terms are administered to participants to find out how people 
actually use them in every day descriptions of persons and then we take a look at 
which ones are redundant with each other and generate factors that can 
correlate with each other and get factors that way, of these studies, I said 14 of 
them, 10 of them have been studies of languages of European origin, which does 
have a bit of ethnocentric bias. I am engaged in some project to try to correct for 
that by studying particular African languages, to bring a kind of more global 
diversity to this. 


104 




Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Volume 2, Part 2: Related Constructs 


What We Learn From Lexical Studies of 
'Indigenous' Personality Factors 

(factor analyses of very large #s of descriptors) 

o "Big One" solutions yield an Evaluation 
(desirable versus undesirable traits) factor 

o "Big Two" solutions yield Social Propriety 
(socialization) and Dynamism factors (the 
saint, the hero) 

o Five-factor solutions most often yield the 
Big Five, but not always 

o Seven-factor solutions quite often 
(Filipino, Hebrew, and others) show an 
alternative "Multi-Language Seven" pattern... 


But what I want to do here is summarize fairly quickly some extensive 
reviews of this literature that I’ve done with Lou Goldberg, these reviews have 
been published in a succession of books and journals. What you get really 
depends on how many factors you extract. If you just extract one factor which of 
course would be unrotated the universal pattern in all these studies, most of them 
have reported this is that kind of a Big 1 solution would be an evaluation factor, 
contracting desirable with undesirable attributes. That’s also incidentally the Big 1 
factor in Osgood’s studies of judgments about the attributes about objects in 
general so there is probably a larger cognitive architecture going here. 

A Big 2 solution which would be two rotated factors, this is also never 
been disconfirmed in any studies and most of them have reported something like 
this. That one of the factors will be social propriety, another could be labeled 
dynamism, social propriety would include things like, being polite and responsible 
and humble and patient, where as dynamism is more about being stimulating and 
interesting whereas social propriety is about morality, dynamism is essentially 
independent of morality. 
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What We Learn From Lexical Studies of 
'Indigenous' Personality Factors 

(factor analyses of very large #s of descriptors) 

o "Big One" solutions yield an Evaluation 
(desirable versus undesirable traits) factor 

o "Big Two" solutions yield Social Propriety 
(socialization) and Dynamism factors (the 
saint, the hero) 

o Five-factor solutions most often yield the 
Big Five, but not always 

o Seven-factor solutions quite often 
(Filipino, Hebrew, and others) show an 
alternative "Multi-Language Seven" pattern... 


Three and four factor solutions vary a great deal, actually even five factor 
solutions vary a good deal, but you do often find a Big 5 pattern and I’m going to 
assume with this audience that you know what the Big 5 is, but I will review what 
those factors are in just moment, the Big 5 has been found in these lexical 
studies, more often in languages of Northern European origin. I do want to point 
out if you take a Big 5 measure and impose it on some other culture by 
translating the measure, you can get the factor structure out of the other 
language. 

In lexical studies, we studied each language on its own and looked for the 
indigenous structure, this is actually a more stringent cross cultural replicability 
criteria and the Big 5 did not fair as well using that more stringent criteria. Some 
recent studies are indicating that there are structures with more than five factors 
that are probably at least as replicable in these lexical studies as the Big 5, one 
that I’ve published is one that I called the multi-language 7, which I noticed from 
correspondence between Philippino and Hebrew lexical studies and that is in 
2003 European journal of Personality article. 
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Multi-Language 7 - Big Five Factors 

Gregariousness | 

|-> Extraversion 
Self-Assurance | | 

|-> Emotional Stability 

Even Temper | | 

/-> Agreeableness 
Concern for Others| 

Conscientiousness -> Conscientiousness 

Intellect -> Intellect 

Negative Valence . 

(Social 

Unacceptability) 


That particular structure resembles the Big 5 in some ways and is distinct 
in other ways; they both have conscientiousness and intellect factors. The multi¬ 
language seven has a negative valiance factor which has a number of socially 
unacceptable attributes correlating together with it, attributes like evil, good for 
nothing, retarded and corrupt and things like this. 

So I have the Big 5 factors on the right and the multi language 7 on the 
left. It also divides these three kinds of affect interpersonal factors from the Big 5 
somewhat differently into 4 factors. So that you get gregariousness and self 
assurance, even temper, concern for others, this is supposed to represent that 
the extraversion variance in this model is split into gregariousness and self 
assurance, emotional stability is split into self assurance and even temper which 
is kind of lacking a bad temper, so you have fear and anger actually separated in 
this model on the left. And agreeableness is split into even temper and concern 
for others. Another approach we can take in looking at content representation 
and of personality measures is to look at what source of personality variables 
might be well outside the Big 5. 
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What Traits (Represented in 
Adjectives) Are "Beyond the Big 
Five" (Saucier & Goldberg, 1998)? 

...of 53 candidate outlier clusters, among 
the most reliable AND orthogonal were: 

* Height (tall-short) 

* Girth (chubby-skinny) 

* Youthfulness (young-old) 

* Religiousness (religious-nonreligious) 

...ideology or trait? 


With Goldberg, I did a study, published in 1998, where we looked at things 
that were really highly orthogonal to the Big 5. That devised 53 candidate 
clusters by studying a variety of adjectives, and the ones that were most, with the 
clusters that were not only orthogonal but also reliable, of course anything that 
was unreliable might also be orthogonal, but it wouldn’t be of any importance. I 
have listed here the four that really was the best case for. One of them was 
height, which there are adjectives for, girth was independent of the Big 5, 
youthfulness and religiousness was virtually independent of the Big 5. Now this 
issue of religiousness as a trait started to raise some issues certainly for me 
about whether this was a personality variable or not, it is a matter of some 
controversy, whether this is ideology or personality. 
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Personality and Ideological Beliefs 
(i.e., social attitudes) 

* Typically separated domains of variables 

* As expressed in adulthood, both are 

substantially heritable; beliefs show 
more shared family environment effects 

* Belief-related dispositions have powerful 
social effects (e.g. aggression/violence) 
and may be just as important 

* Not clear how much overlap between domains 

exists (in popular models and measures) 


So in the next part of my talk, I’m actually going to go off in the direction to 
look at ideology variables and personality variables and how they compare to 
each other and how much they overlap. Traditionally social attitudes and 
personality have been separated. I’m not quite sure what to call this second 
domain, social attitudes, I don’t like the title so much, ideological beliefs, I think it 
implies something, I don’t think that’s quite the optimal, if I had to devise the 
perfect label for it, it would be beliefs about what is true, or real, or good. But 
that’s kind of long label, I’m still working on the best label for this domain. But I’m 
going to use these labels for it today, these have been typically been regarded as 
separable domains and one reason its long been assumed that the ideological 
belief or social attitude domain would not show inheritability would have strong 
environmental influence. But in fact the behavior genetic studies don’t completely 
support that, they do show more of the tendency for belief to have shared 
environmental effects, at least in youth, not necessarily later in adult hood. But 
both are actually substantially heritable. 

Of course a wide range of beliefs have not been studied yet. I’m going to 
use an extremely broad measure of ideologically beliefs later on here. I believe 
that most of those constructs I think heritability has not really been directly 
studied yet. This position that leads to beliefs are important variables that have 
powerful social effects. So I don’t think they should be summarily disregarded. 

But it’s not clear just how much overlap there is between these two domains, so I 
want to exert some things from a current paper of mine on this relationship. 
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Ideological Beliefs 
(Social Attitudes) 

Defining them: "Representations 
about what is true or real or 
good, held by an individual, 
though capable of guiding a group" 

Measuring them: There is no 

comprehensive inventory, indeed 
often represented simplistically 
as a single continuum 


First of all, I’m going to have to define what I mean by ideological beliefs 
or social attitudes and this is my definition. Representations about what is true or 
real or good held by an individual though capable of guiding a group, this 
definition is my own but it is based on a rather detailed survey of definitions in the 
psychological and sociological literature. In terms of the measurement of 
ideological beliefs I would say that the situation is somewhat more primitive that 
as the case for personality. There is no comprehensive inventory and indeed the 
domain is often assumed or thought to be represented by a single continuum, 
perhaps the left right, or kind of political and religious kind of continuum. 
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Lexical Rationale Applied to Ideological 
Beliefs: Saucier's (2000) Study of 
English-language "-ism" Terms 

• Assumed the most important concepts in 
domain represented in words with -ism 
ending 

• Extracted 266 such terms from dictionary 

• Built items from content in the definitions 

• Constructed 400-item inventory 

• Administered to 500 university students 
(CA) 

• Found internally replicable 3 and 4 factor 
structures 

• In follow-up studies, the 4-factor 
structure has proved far more replicable 


In order to kind of get beyond those kinds of assumptions, in a paper 
published in 2000 in JPSP, I reported on a study of isms terms in the English 
Language. That was just in one language, but the results have been well 
replicated in a yet so far unpublished study in Romania, done by Stephen Kraus. 
This was based on the idea that for the beliefs ideological belief domain that you 
could capture the most important variables by looking at words that end in I S M, 
ism. 


So what I did with my research team is to go through the American 
Heritage Dictionary of the English Language, we extracted 266 isms terms that 
did refer to beliefs, now we did have to leave out some terms that end in ism 
such as metabolism or alcoholism, I don’t want to short strip those things, I’m a 
strong believer in metabolism but it wasn’t really a kind of belief that there would 
be individual differences in, so we were concentration on ones that were actually 
belief terms. The strategy here was not unlike the lexical studies of personality, 
we did not administer the actual ism term to two people, partly because some of 
them are a bit unfamiliar and also we expected a tendency where many people 
would say that I don’t believe in any isms, I believe what is true they are not isms 
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Lexical Rationale Applied to Ideological 
Beliefs: Saucier's (2000) Study of 
English-language "-ism" Terms 

• Assumed the most important concepts in 
domain represented in words with -ism 
ending 

• Extracted 266 such terms from dictionary 

• Built items from content in the definitions 

• Constructed 400-item inventory 

• Administered to 500 university students 
(CA) 

• Found internally replicable 3 and 4 factor 
structures 

• In follow-up studies, the 4-factor 
structure has proved far more replicable 


But what we did instead is that we looked at the dictionary definition and 
we turned the definition into questionnaire items. For example a hedonism item 
was definition included the idea that pleasure is the chief goal of life, or one 
definition was the belief that pleasure is the chief goal of life, we just made it into 
an item, pleasure is the chief goal of life; do you agree or disagree? By a direct a 
method as possible we just converted the definition into an item. Out of that we 
constructed a 400 item inventory, it’s more than 266 because we had to, many of 
the definitions were multi, there was more than one definition for the term and 
some of the definitions were so long we had to split that we had to split them into 
multiple items then recombine them later on. 

A good example of that is environmentalism, which has two very different 
meanings, environmentalism can mean you want to protect the environment from 
pollution, or it could mean that you believe that environment is more important 
than genetics. Those two are completely separable forms of environmentalism. 

So we administered the 400 item inventory to 500 university students and 
found an internally replicable 4 factor structure, replicable across halves of the 
sample. And in follow up studies I compared various structures from the initial 
study and it was the 4 factor structure that proved the most replicable. 
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Four Replicated Isms Factors 

Alpha (a) -- Religious Orthodoxy (encompassing 

fundamentalism versus secularism)... religiousness, 
correlated with conservatism and authoritarianism 

Beta ((3) - Unmitigated Self-Interest - advocating 

unfettered exploitation of opportunities by self or 
close in-group... correlated with materialism, 
hedonism, ethnocentrism 

Gamma (y) - Protection of Civil Order (status quo 

polity, common civil institutions) 

Delta (5) - Unorthodox Spirituality and Mysticism - 

emphasizes subjective experiences, including 
paranormal ones 


So let me just review what these factors are so you know what the content 
of some the later analysis are going to be dealing with. My initial labels for these 
were cautious, and I used Greek letters because I didn’t want to commit myself 
too early to the meaning to labeling and this was partly informed by the 
experience with the Big 5 where I think some of the labels are not so great. But I 
would characterize the first biggest factor as religious orthodox, which I call the 
alpha factor. It does involve religiousness and it encompasses a stark contrast 
between fundamentalism on one pole and secularism on the other and is 
correlated as I demonstrated in the 2000 study with measures with conservatism 
and authoritarianism actually quite very substantially. In fact actually if we go 
back to this factor, this is the factor that has been most emphasized in previous 
measures of social attitudes. So that if you had put all of them together you 
would probably get one big factor which was this and you wouldn’t get so much 
as the next three factors that I’m going to subscribe, one of which is I call Beta, it 
could be labeled as unmitigated self interest, a belief that you should pursue your 
self interest sort of no matter what versus belief in the limit on self interest, so 
through the core of logic through the high Beta person would be that they 
advocate unfettered exploitation in environmental opportunities by their self or by 
self close end group such as their nation or their family against others and some 
of the correlates are actually components of this factor are materialism, 
hedonism, and ethnocentrism, from an evolutionary standpoint, this is probably 
maximizing short term fitness but disregarding considerations of long term 
fitness. 
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Four Replicated Isms Factors 

Alpha (a) -- Religious Orthodoxy (encompassing 

fundamentalism versus secularism)... religiousness, 
correlated with conservatism and authoritarianism 

Beta ((3) - Unmitigated Self-Interest - advocating 

unfettered exploitation of opportunities by self or 
close in-group... correlated with materialism, 
hedonism, ethnocentrism 

Gamma (y) - Protection of Civil Order (status quo 

polity, common civil institutions) 

Delta (5) - Unorthodox Spirituality and Mysticism - 

emphasizes subjective experiences, including 
paranormal ones 


A third factor labeled gamma really refers to beliefs in the civil rather than 
a religious kind of order, almost a civil religion to use the term of Russo, so 
protection of civil order is the emphasis here, and that is the status order quo 
political orientation and a common civil institutions like the government and the 
constitution and so on. And the fourth factor is Delta which is unorthodox, 
spirituality and mysticism as contrasted with the more orthodox religiousness of 
the Alpha factor, the emphasis here is one subjective experiences, rather than 
kind of an extrinsic approach to religion and it includes a tendency to believe in 
paranormal experiences as well. 
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Derivation of public-domain Survey 
of Dictionary-based Isms (SDI) 

• Delineates lower-level constructs, not just 
broad factors, each an item composite with 
~4 items, keying in both directions 

• Each based on at least two of the original 
400 isms items 

• Created new items to allow balanced keying 

• Administered to new college-student sample 
(N=383, OR) 

• Finalized using data from third, community 
sample (N=703, OR) 

• In end: 48 item-composites, 12 assigned to 
each factor, a .51 to .91 (half above .73) 

• For each factor, primary (p; more univocal) 
and secondary (s; less univocal) subscales 


In order to develop better measurement in this domain, I wanted to 
develop a measure that had a lot of fidelity with regard to lower level constructs, 
not just broad factors. So I constructed a large number of item composites that 
have roughly four items, and I found it very important to make the direction of 
keying balanced within these item composites because attitude measures have a 
rather troubled history of being affected by acquiescence individual differences in 
acquiescence, and I wanted to get a control on that. Each of the item composites 
are based on at least of the two original 400 ism items, but I did generate 
additional items in order to allow for balance keying, that was necessary. So I call 
this the survey of dictionary based isms, it is a public domain inventory, not 
proprietary. I administered the initial draft of this to new college students sample 
and finalized it with a third community sample. Has 48 item composites, 12 
assigned to each factor with Alphas between .51 and .91 and I do divide these 
subscales into primary and secondary subscale based on their tendency to be 
always loading less highly on the factor versus shifting around from one factor to 
another. 
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Empirical Study: Relation of 
Personality and Ideological Beliefs 

2000 study indicated isms quite independent of 
adjectival Big Five — but what about other 
personality models? 

METHOD 

N=703, Eugene-Springfield community sample, 57% 
female, average age 59 (range 20s to 90s) 

Administered 'Survey of Beliefs and Opinions' 
including the SDI items 

Had been administered numerous personality 
inventories in previous eight years 


I have a current paper that I’m finishing off about the relation between 
personality and ideology beliefs in the 2000 study I found that the isms were 
quite independent of the adjective Big 5, that was just one brief measure that I 
used in that study. I am going to describe here a much more involved study 
looking at a large variety of personality measures, using the Eugene Springfield 
community sample. And I administered to this sample a survey of belief and 
opinions which included all of the SDI items that make up those 48 clusters. This 
sample had been administered numerous personality inventories in the previous 
8 years so it enabled some useful comparisons. 
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Personality Inventories Administered 


* 10-adjective measures for each factor: 

Big One (Evaluation) 

Big Two Dynamism 

Big Two Social Propriety 

* 100 Big Five marker adjectives (of 
Goldberg) 

* 60-adjective marker set for the Multi- 
Language Seven 


I didn’t look just at the big five with regards to adjectives; I also used 10 
adjective measures that I used in another published study for the Big 1 and for 
the Big 2 factors that are lexical. I used Goldberg’s hundred Big 5 marker 
adjectives for the Big 5, and I used the 60 adjective marker set for the multi 
language 7. 
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Personality Inventories(and # of scales) 

California Psychological Inventory (CPI; 3) 

NEO Personality Inventory (NEO-PI-R; 5) 

Six-Factor Personality Questionnaire (6FPQ) 

Hogan Personality Inventory (HPI; 7) 

Temperament and Character Inventory (TCI; 7) 

Multidimensional Personality Questionnaire 
(MPQ; 11) 

16 Personality Factors (16PF) questionnaire 


In addition, I looked at 7 different well known personality inventories, and I 
used broad at least fairly independent factors that can be derived from these 
measures. So that Harrison Gough CPI I used the 3 Factor scales, for the NEO 
inventory of I used of course the 5 domain scales, Doug Jackson 6 factor 
personality questionnaire I used these 6 broad scales, Hogan personality 
inventory, I used 7 broad scales, Cloninger’s temperament and character 
inventory I used the 7 broad scales, for Tellegen’s MPQ, I used the 11 Broad 
scales, these are getting to be a little bit correlated, but I kept them, kept all 11 of 
those, and with the 16 PF I used all 16 partly because that is the most commonly 
used scales, although there are broad scales for the 16 PF, to measure the isms 
factors I used the 48 SDI item composites and generated factor scores from 
them. 
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Isms Factors from 48 SDI Item- 
composites 

• Factor scores from four-factor varimax- 
rotated solution (principal-factors 
extraction method) 

• Eigenvalues consistent with optimal 4- 
factor solution 

• All primary (p) subscales loaded on 
expected factor 

• Reasonable representation of 4 isms factors 


The Eigenvalues from that factor analysis were consistent with a 4 factor 
solution being optimal. And all of my P subscales or the primary subscales 
loaded on the expected factor. And I do think this is a reasonable representation 
of the 4 isms factors. 
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Single Bipolar Continuum of 
Ideological Beliefs: 
Republican versus Democratic 

Items: 

"Politically I 
favor the 
Republican party" 
and "Politically 
I favor the 
Democratic party" 

• Second response 
subtracted from 
first 

• An a of .93 


Now I mentioned before that there are often ideological beliefs are 
assumed to be represented by a singular bipolar continuum which is in America 
would correspond to Republican and Democratic Party. So I did have two items 
on my inventory that refer to this: “Politically I favor the Republican Party” and 
another one for the Democratic Party. I simply had a simple measure of this by 
subtracting response of the second from the first, this is all using a 1 to 5 multi 
point rating scale and it turns out they are very highly correlated around negative 
.8 I believe, and the Alpha was .93 for only a 2 item scale, pretty reasonable way 
of measuring this. 
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Analysis Strategy 

* Each ideological belief variable 

- American Political Party Affiliation 

- 4 isms factors 

served as criterion variable in multiple 
regression 

* The set of scales on each inventory served 
as a set of predictors in each regression 
procedure 

* Adjusted (shrunken) R values examined - 

corrects for different number of 
predictors and sample-size variations 


So I am going to be comparing that particular single bipolar approach to 
ideological beliefs with the isms factors here, so my analytic strategy here was to 
use multiple regression for each of the ideological belief variables that would be 
the Republican minus Democratic measure and each of the four isms factor was 
a criterion variable in multiple regression. And the predictors for each regression 
were all the scales from one of the personality inventories, for example all of the 
MVQ scales would be the predictors and one of the ideological belief dimensions 
here would be the criteria, I’m going to look at how big the multiple R’s are, but I 
am going to use adjusted or shrunken R values because I’ve got different 
numbers of predictors with these different inventories and also because I’m using 
deletion procedures I’m getting some sample size variations and I wanted to 
make everything more comparable. 
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Adjusted R for Lexical Personality Models 
Predicting Ideological Dimensions 


Ideological Dimension 

Bigl 

Big2 

Big5 

ML7 

Republican - 

Democratic 

.00 

.08 

.28 

.34 

Alpha (Relig. 

Orthodoxy) 

.00 

.04 

.25 

.40 

Beta (Self-interest) 

.14 

.21 

.27 

.23 

Gamma (Prot. 

Civil Order) 

i .23 

.15 

.33 

.41 

Delta (Unorth 

. Spirit.) 

.06 

. 12 

.31 

.31 


Here are the adjusted R values that I get in this analysis for using the 
person the lexically derived personality measure, these are adjective measures 
in predicting the ideological dimensions. You can see that the Big 1 in some 
cases has a zero correlation, it doesn’t predict some of the ideological 
dimensions at all. Generally as you add more factors to the lexical model, you get 
increased prediction of the ideological dimensions. But none of these coefficients 
is the highest is .41. 
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Adjusted R for Questionnaire Models 
Predicting Ideological Dimensions 


Ideol.Dim. 

CPI 

NEO 

6FPQ 

HPI 

TCI 

MPQ 

16PF 

Rep.-Dem. 

.26 

.39 

.24 

.20 

.25 

.46 

.39 

Alpha 

.36 

.47 

.32 

.33 

.63 

.65 

.55 

Beta 

.33 

.33 

.20 

.20 

.50 

.30 

.53 

Gamma 

.38 

.36 

.33 

.38 

.38 

.46 

.48 

Delta 

.09 

.44 

.19 

.29 

.44 

.48 

.31 


If I then turn to the questionnaire models I start finding many of the 
multiple R’s are now exceeding .41, in fact about a third of them are here, so we 
do get higher multiple R values even though we are using shrunken R values, 
with the questionnaire model, so that is true for some questionnaires and not for 
others, the CPI and the 6FPQ and the HPI really don’t generate higher values 
than the lexical models do, but the NEO, the TCI, MBQ and 16PF all generate 
higher multiple R values. That indicates that they have more overlap with 
ideological beliefs than the other measures do. But something’s going on here 
that’s making these measures really encroach into the ideological belief domain, 
more than other personality measures. And it’s more true if you notice here for 
the Alpha factor. That’s true, the highest always for the Alpha factor for all of 
those four inventories that I targeted, however there are some substantial 
multiple R’s for the other ideological dimensions as well. I was somewhat 
surprised to find that you could predict voting preferences probably to some 
degree using personality measures. 
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Summary of Multiple Regression Results 

Coefficients in table are higher for 
questionnaires, especially 16PF, MPQ, 
TCI, and NEO-PI-R 

Tendency - though not monotonic - 
more scales in inventory, higher R for 
ideological belief dimension 

Religious orthodoxy especially well- 
represented in these inventories 
MPQ Traditionalism 
TCI Self-Transcendence 
16PF Rule-consciousness 
NEO-PI-R Openness (low) 


So I think I’ve summarized this already that the coefficients are higher for 
questionnaires especially for the 16PF, MPQ, TCI, and NEO, in the case of the 
NEO it’s really the openness to experience scale, which is generating this, 
because it has the same pattern of correlation with the isms factors as that 
republican democratic measure did. That reflects the fact that high openness to 
experience tends to predict to almost R (square) = .4 that one will vote 
Democratic, as opposed to Republican. Religious orthodoxy is especially well 
represented in these inventories but to a lesser degree but we do find other 
dimensions represented. 
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Correlations of Specific Personality 
Scales with Isms Factors 

• Most correlations low, but highest are: 

ALPHA (Relig. Orthodoxy) 

.63 MPQ Traditionalism 
.60 TCI Self-Transcendence 
.45 16PF Rule-consciousness 

BETA (Unmitigated Self-Interest) 

-.41 TCI Cooperation 

GAMMA (Protection of Civil Order) 

.39 CPI Norm-favoring 

DELTA (Unorthodox Spirituality) 

.44 TCI Self-Transcendence 
.42 MPQ Absorption 


If you look at the specific personality scales that are generating these 
correlations we see that MPQ has a traditionalism scale TCI has a self 
transcendent scale, 16PF has a real consciousness scale those are really kicking 
those multiple R’s up quite a bit. NEO openness is actually correlated just below 
these levels with all of the factors but especially with alpha. You also see 
cooperation, Norm favoring and absorption scales being correlated with the isms 
factors. 
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Personality and Ideology: Are They 
One Thing, or Two? 

• Overlap is modest for the two-party continuum 

• Higher when four isms dimensions considered 

• Especially high for Religious Orthodoxy (a) 

• Overlap modest for lexical-based scales 

• But higher for questionnaire models that 
include scales like Traditionalism, Self- 
Transcendence, Openness, etc. 

• To a degree, these domains already intertwined 


So I can ask the question of whether personality and ideology are one 
thing or two and you would have to say they tend to be two things more if you 
think about the two party continuum it is somewhat higher if you take into account 
all four isms dimensions, there is more overlap with religious orthodoxy, so you 
get that kind of content encroaching into personality measures more than other 
kinds of contents, lexically based scales have rather less overlap and so to a 
degree these domains are already intertwined and you use a lot of the 
personality measures you’re already involuntarily perhaps measuring ideological 
beliefs to a degree. 


126 




Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Volume 2, Part 2: Related Constructs 


Do Isms Factors Offer Incremental 
Validity Over Personality Factors? 

•Isms factors largely independent of lexical 
personality factors 

• But this does not establish their importance - 
their validity 

• Previous literature (correlated constructs) 
already establishes good case for Alpha (Relig. 
Orthodoxy), less so for other factors 

• An informative criterion test would be with 
measures relevant to "emotional intelligence" 


Now there is a question of whether these ideological beliefs are a 
nuisance variables or whether they actually add something interesting to 
prediction of behavior. This is what I want to get to next, and this will bring us 
more directly into emotional intelligence topic. Largely the isms factors are 
independent of the lexicon based personality factors. But just because they are 
independent of them doesn’t mean that they’re somewhat useful beyond those 
factors. There is already a pretty good case, that for constructs associated with 
alpha associated with Alpha such as authoritarianism or conservatism that those 
are useful variables for some things, we don’t much about the validity of other 
factors. So what I want to use here as the last part of my talk is a criterion test 
where I use self report measures that are supposed to be relevant to Emotional 
Intelligence and I am going to hit personality measures against these ideological 
belief measures and see what if the beliefs actually add to the predictions over 
personality measures. 
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Criterion Self-Report Measures 

Toronto Alexithymia Scale (20 items) 
...used three subscales... 

Externally-Oriented Thinking 
Difficulty Identifying Feelings 
Difficulty Describing Feelings 

Barchard's Emotional Intelligence scales 

Attending to One's Own Emotions 
Negative Expressivity 
Positive Expressivity 
Responsive Joy 
Responsive Distress 
Empathic Concern 
Emotion-based Decisionmaking 


The Emotional Intelligence relevant self report measures fall into two 
groups, one is the Toronto Alexithymia Scale which I’m using the 20 item version, 
and I’m going to use not the overall score, but the three subscales scores, 
externally oriented thinking, difficulty identifying feelings and difficulty describing 
feelings, also Kim Barchard in her Doctoral Dissertation devised some Emotional 
Intelligence self report scales. I don’t want to represent that she thinks that this is 
the best measure of Emotional Intelligence, actually she concluded Cognitive 
Measures that treated as ability would be superior, these she thought had too 
much overlap with personality. But these are the scales that she used, Tending 
to Ones Owns Emotion, Negative Expressivities, Positive Expressivities, 
Responsive Joy, and Distress and Empathic Concern and Emotion Based 
Decision Making. Some of these are related to some of the emotional skills that 
Paul Ekman talked about, such as being able to identify feelings, and paying 
attention to your emotions, although this isn’t self report format these are 
conceivably relevant to emotional skills. 
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Method for Incremental Validity Study 

• 522 community-sample participants completed 
all measures 

• Lexical personality scales added as blocks 
with increasing numbers of factors, to examine 
their relative incremental validity: 

Big One -> Big Two -> Big Five -> ML7 

• Final block: the four isms factors 

• Examined change in R-squared values for each 
new block of variables, for each of 10 EI- 
relevant measures 


My method here was to use 522 participants from that community sample 
and to actually in addition to comparing the attitudes with the personality skills I 
am going to look at how adding more lexical personality factors add to the 
prediction to what degree it adds to the prediction to emotional intelligence, so 
I’m really comparing Big 1, Big 2, Big 5, and multi language 7 representations of 
predicting self report emotional intelligence measures. And the final block is 
going to be the four isms factors, so I’m going to be capturing the change in our 
square that you get from adding the isms factors after the personality variance 
has already been taken out. I am going to be looking here at change in our 
square values for each of the 10 El relevant scales separately, these are the R 
squared change values I get and the * here indicated it was significantly change 
in our R-square. 
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Hierarchical Regression Results 


TAS 

Bigl 

Big2 

Big5 

ML 7 

Isms 

Ext.Oriented 

.01* 

.07* 

. 12* 

.04* 

.11* 

Diff.Id.Figs. 

. 07* 

.03* 

.13* 

.04* 

.03* 

Diff.Des.Figs. 

. 06* 

.16* 

.05* 

. 04* 

.02* 

Barchard scales 

Emot.Decisions 

.00 

.03* 

. 14* 

.05* 

. 13* 

Empathic Concern 

.05* 

.06* 

.10* 

.04* 

.10* 

Attending Emots. 

.00 

. 02* 

. 17* 

.03* 

.06* 

Responsive Joy 

.10* 

.05* 

. 12* 

.08* 

. 04* 

Respon. Distress 

.00 

. 02* 

.22* 

.07* 

. 04* 

Positive Exprvty 

.08* 

. 11* 

. 12* 

. 06* 

. 04* 

Negative Exprvty 

.00 

. 13* 

.11* 

.05* 

. 02* 


And it turns out that adding more personality factors in essentially every 
case adds significantly to the prediction of these El measures; adding the isms 
factors also in every case added significantly to their prediction. You can get 
more prediction by adding more personality factors but also the isms were 
adding, but I do want to point out that the change in R squared is especially large 
for three of the El measures, but it is relatively low for some of the others. The 
externally oriented thinking isms actually added quite a bit to the prediction of 
that, also emotional decision making and empathic concern. Overall the 
personality variables the multiple R was .53 and adding the isms factors raised 
this value to .58. If you turn the analysis around enter the isms factor first they 
generate a average multiple R across these scales of .37 which is lower but for 
those three highest coefficients of externally oriented emotional decision making 
and empathic concern, actually isms on their own predicted the El scales more 
highly than the personality measures did. 
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Results of Hierarchical Regressions 
Typically... 

~11% of El-scale variance accounted for with 
only one or two-factor personality models 

Adding Big Five more than doubled the R 2 
Adding Multi-Language 7: Further improvements 

Adding isms factors always significantly 
increased R, with 2 to 13% change in R 2 

This increase highest for El scales involving 
attention, preoccupation, decision processes 


You can get about 11% of the El scales variants with the 1 or 2 factor 
personality models adding the Big 5 more than doubled the R squared, it 
increased beyond 22% and the multi language 7 would add significant further 
improvements, the isms factor always significantly increased the R but how much 
varied from one scale to another. And the ones where it did have more of an 
effect related to attention preoccupation and decision making which may be more 
cognitive aspects of El. 
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Isms Factors and El Scales 

• Beta (unmitigated self-interest) had 
significant negative association with 
all ten El scales 

• Delta (unorthodox spirituality) had sig. 
positive association with 7/10 

• Gamma (protection of civil order) sig. 
associated with 7/10 

• Alpha (religious orthodoxy) predicted 
none...Alpha does predict other things! 


I took a look at which of the isms factors was actually contributing to this 
improvement and prediction of the El scales. And it was never the Alpha Factor 
that was doing so. It was in fact the Beta factor was doing it in every single case, 
all ten scales, it had a significant Beta coefficient. The Delta in 7 out of 10, the 
Gamma in 7 out of 10, and the way it goes is that high Beta belief and 
unmitigated self interest tended to be correlated with lower scores on the 
emotional intelligence measures. 
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Two SDI Scales Frequently Adding to 
Prediction of El Scales 

Delta pi: Intuitionism 

Example item: "Enlightenment can be gained 
through meditation, self-contemplation, and 
intuition" 

Beta si: Physicalism 

Example (reverse-keyed) item: "Physical laws 
cannot explain some mental phenomena" 

...Both represent positive (versus indifferent) 
valuation of internally-oriented thinking 


I wanted to go to even a more detail so this the result of step-wise 
regression where I looked at which of the item composites were actually most 
contributing to the prediction of the El measures, and these are the two that 
stood out, so when there was a high increment of prediction these were always in 
the equation, pretty strongly. One of them was the intuitionism scale an example 
item, a belief that you can gain enlightenment through meditation, self 
contemplation and tuition. The other one is physicalism scale which would be 
choosing a reverse keyed item, because the reverse keyed item that would be 
the one that beta 1 would be positively correlated with the El scales, belief that 
physical laws cannot explain some mental phenomena this actually helpful at 
least to me in understanding what is going on here because both of these 
represent a valuing of internal oriented thinking as opposed to only focusing on 
external. It may be that emotional intelligence; one contributory or precondition is 
the belief that subjective experience and what’s going on internally is important. 
And that is not going to be captured on personality measures. 
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Important Caveat! 


The "emotional intelligence relevant" 
scales employed here were all self- 
report, and not maximum performance 
tests. 


One caveat is that these are just self report measures not maximum 
performance tasks so they subject to all the limitations to such measures. 
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Overall Conclusions 

• The Big Five doesn't provide a comprehensive 
account of individual differences 

• Lexicon-based personality dimensions (like the 
Big Five) are generally independent of ideological 
beliefs 

• Questionnaire models of personality often 
contain certain scales rather saturated with 
ideological-belief variation 

• Ideological-belief dimensions (represented by 
isms factors), added to personality measures, 
increase the prediction of important criteria 
(including some related to El) 


The Big 5 does not provide a comprehensive account of individual 
differences, I think I’ve given some demonstration of that although Lexicon based 
personality dimensions are generally independent of ideology beliefs the 
questionnaire models are not always so and the ideological belief dimensions 
when you add them to personality measures increase the prediction of important 
criteria, including some that are related to emotional intelligence so one of the 
contributions that this may make to the literature on emotional intelligence is just 
indicating that you can’t disregard the importance of beliefs. Peoples beliefs 
systems in terms of how they handle their emotions. 
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I just want to acknowledge research assistance by five key people, and 
also the grant support from the National Institute of Health. Thank you very 
much. 
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DISCUSSION 


DISCUSSION 

DR. KYLLONEN: We have time for some questions. 

DR. SCHERER: One of the things that I was particularly intrigued by is the 
possibility that the isms factors that you identified might play a role in some of the 
biases and appraisals that I tried to describe yesterday, particularly paranormal 
beliefs in terms of, for example, attribution of responsibility, particularly in terms 
of causal responsibility. Do you think that there is any evidence that you are 
aware of, or do you have any hypotheses, concerning the effect on such 
appraisal factors? 

DR. SAUCIER: No, I am not aware of such evidence, I think there has 
been a dearth of studies in this area; ideological beliefs have been separated off 
into a domain for political psychologists and sociologists maybe, so we haven’t 
really looked at that too much. Just from the content of the measures you would 
expect a good bit of that. 

DR. SCHERER: We did a little study where we tried to measure 
paranormal beliefs in 3 factors, one was divination, and people believe in 
divination practices, they actually think that there is less chance in appraisal. A 
little bit of evidence that it might work, could be very interesting. 
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DISCUSSION 


DR. JOHNSON: What words happen to have isms in them and to what 
extent are they a happenstance of the English Language? What likelihood do you 
think there is that you are overlooking important belief systems by focusing on 
these words? 

DR. SAUCIER: There is a good deal of likelihood that I’m overlooking the 
belief system, but I am correcting for that, because what I’ve done is I did a 
survey of the literature and I have 42 additional item composites that were 
actually administered to the same sample. I just haven’t analyzed that data as 
much, the full comprehensive measurement that I would have right now would 
have ninety item composites, and that includes things like social dominance 
orientation which is not terribly well captured there. I have an authoritarism scale, 

I have some things related to attribution, why poor people are poor, lots of other 
variables like this. I am making an attempt to come up with something that fills in 
the holes. 

DR. KYLLONEN: Another question. The question is there are a lot of 
social attitude surveys out there and maybe when you responded to Dr 
Johnson’s question you were saying that is what you were doing in the 
supplementary measures. I guess my question is how much do the isms factors 
capture what the existing social attitude scales survey, and these dimensions 
already say? 
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DISCUSSION 


DR. SAUCIER: To get a really get a good answer for that, I would have to 
administer all those social attitude measures, which I haven’t really had the 
resources to do. What I did is this: I looked at those measures and figured out 
what core items were and more or less, adopted items like that into these item 
composites. But I haven’t looked at that directly. I can say that if you factor 
analyze all 90 item composites -- the isms ones plus the additional ones -- you 
can get a couple of additional factors that are fairly independent of the first four, 
one of which has to do with belief whether big government is good or helpful, and 
another has to do with whether you want to treat outsiders in society harshly or 
kindly. So you do get some other kinds of content that way. 

DR. KYLLONEN: One other question. You mentioned the first factor was 
very important relative to the others, indeed you suggested it was larger. How 
much larger? 

DR. SAUCIER: It’s not hugely larger, it’s every factor analysis I’ve done, 
it’s always been the first one. But it’s not twice as large, even looking at the 
unrotated Eigenvalue; it is never more than twice as large as the second. 
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DISCUSSION 


DR. SIGEL: You mentioned that these belief systems are what you are 
getting inside ideology. How do they relate to any social behavior, for example, 
voting behavior? Do you find any relationship between them? 

DR. SAUCIER: Social behavior for example which? 

DR. SIGEL: Voting behavior or other kinds of thing? 

DR. SAUCIER: Voting behavior, I haven’t personally looked at that 
relationship now, this Republican-Democratic continuum that I came up with 
should predict voting behavior pretty well unless I’m getting a lot of lying. The 
isms factors are correlated, you can a multiple R, I believe around .7 for 
predicting that particular continuum from the isms factors. Now this has raised 
one issue for me, which I’ve considered that it might be useful to have a model at 
least for American purposes, where you have one factor, be that political 
continuum and the other factors are completely independent of that. And it is 
clear that you can get three factors, which are completely orthogonal to that 
political continuum. In that model, one would predict voting behavior very highly 
and the others would maybe predict specific issues but not so much others. 
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DISCUSSION 


DR. SIGEL: A related questions in terms of your notion of ideology and 
ideological beliefs. Is your view on belief a kind of a singular, univariate thing or is 
it componential? How would you define belief in a componential way, because 
there are other factors that are related to it? 

DR. SAUCIER: How would I would I find beliefs in such a way? 

DR. SIGEL: Components? Let me explain. For example, in the parent 
child literature there is a lot of stuff dealing with beliefs very differently than what 
you’re dealing with. For example, a belief is more than just what you’re saying. A 
belief involves also affect, knowledge and so forth, all of which are tied into this 
belief, but you’re using belief primarily as the definition you presented, right? 

DR. SAUCIER: It emphasizes rather broad global beliefs, about what is 
true, real, and good. You can devise beliefs about very specific kinds of entities, 
kind of endlessly and I can’t guarantee that these are going to comprehend 
those, they probably don’t completely. For example, the previous presenter had 
some, had conditional reasoning items, and I was able to make out for most of 
his implicit beliefs of aggressive people, I could come up with explicit belief items 
that I could put on inventory like this. So its’ method that is fairly adaptable but it 
is not comprehensive of all beliefs. I agree! I tend to focus on broad, highly 
generalizable types of beliefs. 

DR. KYLLONEN: Okay, thank you. 
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Social/emotional intelligence: 
From factors to functions 

Maureen O’Sullivan 
University of San jfrancisco 



SOCIAL/EMOTIONAL INTELLIGENCE: FROM FACTORS TO FUNCTIONS 

DR. KYLLONEN: I’d like to now introduce Maureen O’Sullivan, who is 
going to discuss “Social-emotional intelligence: From factors to functions.” 

DR. O’SULLIVAN: A rose by any other name would smell as sweet, but if 
we call emotional intelligence by another name, would it sound as sweet? If we 
called it emotional consensus, or emotional interest, or agreeableness or 
emotional competence, as Klaus suggested yesterday, would it still have that 
zing, which has all of us here this weekend. 
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Definitions 


• Social intelligence: .. .the ability to understand and manage men 
and women, boys and girls-to act wisely in human relations.” 

• Thorndike, 1920, p. 228 

• Behavioral cognition: “.. .the ability to understand the thoughts, 
feelings and intentions of other people insofar as they are 
manifested in discernible behavior.” 

• O’Sullivan & Guilford, 1975 p. 256 

• Emotional intelligence. .the ability to monitor one’s own and others’ 
feelings and emotions, to discriminate among them and to use this 
information to guide one’s thinking and actions” 

• Salovey & Mayer, 1990, p. 189 



Psychologists have been interested in this general area of social, 
emotional, behavioral intelligence for a very long time. For example, I have up 
here some definitions, dating back to 1920, which is the earliest I could find for 
psychologists interested in this general area. 

E. L. Thorndike, said that there were three different kinds of intelligence, 
one of which was social, he disagreed with Klaus Scherer, and thought there was 
such an intelligence. 

In his 1957 structure of intellect model, J. P. Guilford suggested that 30 of 
his 120 factors of intelligence were related to what he called behavioral 
intelligence, the definition on the screen relates to our study of behavioral 
cognition or understanding other people, just a small part of that domain. He had 
hypothesized 24 other intellectual abilities, including behavioral creativity, 
behavioral appropriateness and so forth. 

The last definition is Salovey and Mayer’s original definition of emotional 
intelligence, now certainly there are differences among these definitions and the 
most striking one and perhaps the most important being the inclusion of self 
knowledge in the Salovey-Mayer definition. 
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Definitions 


• Social intelligence: .. .the ability to understand and manage men 
and women, boys and girls-to act wisely in human relations.” 

• Thorndike, 1920, p. 228 

• Behavioral cognition: “.. .the ability to understand the thoughts, 
feelings and intentions of other people insofar as they are 
manifested in discernible behavior.” 

• O’Sullivan & Guilford, 1975 p. 256 

• Emotional intelligence. .the ability to monitor one’s own and others’ 
feelings and emotions, to discriminate among them and to use this 
information to guide one’s thinking and actions” 

• Salovey & Mayer, 1990, p. 189 



But what is so curious is that if we look at how these definitions and how 
they were measured, we would find very little difference among them. 

Thorndike’s 1920 article is illustrated with photographs of a woman posing facial 
expression of emotion. Many of the O’Sullivan and Guilford test assess the ability 
to recognize facial expressions as does many current emotional intelligence 
tests. So although the construct is differently named in terms of measures there 
is considerable overlap in terms of how it is that they are assessed. 

So I find it really curious that we have all these different names and yet we 
have the same way of measuring them. Now certainly there are differences. 
Some use expert opinion, some use consensus, but essentially they are all 
looking at facial expressions of emotions and recognizing them. So different from 
what Bill Revelle talked about yesterday, where with psychometricians you have 
the same name, extroversion, and yet you have very many different ways, and 
everybody knows even though you’re using the same word you’re meaning 
different things. Yesterday when Drew Gitomer opened this conference he noted 
that early attempts to measure social emotional intelligence focused on abstract, 
consideration of factorial uniqueness; early researchers attempted to show that 
there was such a thing as social intelligence and it is still a question for some 
people whether this domain really involves an intelligence, a skill or an ability 
area. 
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So, early researchers were interested in showing factorial separation from 
measures of other intellectual abilities and after many years of trying they did so. 
In 1947, Wedeck, a student of Burt, demonstrated the existence of three factors 
of what he called psychological ability. These factors were orthogonal to two 
other intelligence factors. In 1975 O’Sullivan and Guilford demonstrated.six 
behavioral intelligence factors. The1970’s also saw the publication of several 
tests of the ability to, as they called it then, decode non verbal behavior. But 
these tests were about recognizing facial expressions of emotion, more broadly 
defined. 

These tests of non verbal behavior included Ekman and Friesen’s affect 
recognition test (BART), a precursor of the METT that Paul Ekman talked about 
earlier. Rosenthal’s Profile of Non verbal Sensitivity (the PONS) and Archer and 
Akert’s Social Interpretation Task. None of these tests was significantly 
correlated with verbal or general intelligence. 

So one of our knowns in this area is that it is possible to come up with 
something that relates to social or emotional intelligence, that does not measure 
g. Recent reports of positive correlations aomng modern emotional intelligence 
tests still left us in the dark about whether the shared variance between them is 
due to sensitivity to non verbal behavior, awareness of what the appropriate 
situation is in which to show such behavior, consensual similarities with other test 
takers, or some combination of the three. 
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As long ago as 1955, Cronbach called for the discrimination of what he 
called differential accuracy, differentiating one person from another in terms of 
understanding them, and stereotypic accuracy, understanding commonalities 
among a group of people. The clarification of this distinction in many current 
emotional intelligence tests, I believe, is one of our unknowns. 

In the 1980’s, Sternberg introduced what he called his triarchic theory of 
successful intelligence. One of the most interesting aspects of this theory is its 
underlining the importance of understanding situations and contexts. Sternberg 
and his colleagues developed measures of what he called tacit knowledge; one 
of our colleagues called it common sense yesterday, the ability to understand the 
requirements for success in a particular environments such as the military, the 
business world or the academic world. Data I will present later are consistent with 
this theory, so I think that we can safely say that one of our knowns is that there 
such a thing as tacit knowledge, that people differ in the ability to understand and 
to master to the requirements of different situations. 
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One of the other contributions of Sternberg’s theory is its emphasis of the 
function of intelligence, namely that being intelligent allows one to adapt to be 
successful in achieving one’s goals. This emphasis on measuring functional 
abilities rather than theoretical factors has marked the last 30 years of emotional 
intelligence as Drew noted yesterday. 

What I want to talk with you about today is a program of research on 
individual differences in the ability to detect deception. Detecting deception 
accurately involves understanding the emotional and personality baseline of the 
other person, and being sensitive to emotional and cognitive deviations from that 
baseline. It also involves the willingness to label a person as deceptive. This is a 
motivational and personal attribute that has been little studied. As such I believe 
that the ability to detect deception is one kind of social emotional intelligence. 

This research program was started more than 30 years ago by Paul 
Ekman and Wally Friesen, who set out to study high states lies, because they 
were ones that could cause emotions in liars or truth tellers. They chose 
scenarios and arranged rewards and punishment to exacerbate the emotionality 
of the situation. The first measure of lie detection they devised was a video tape 
test of ten female nursing students, lying or telling the truth about whether they 
are watching nature films, or gruesome surgical films. The young women were 
motivated by their belief that the ability to control their emotion was related to 
their ability to be good nurses, which in fact it was. 
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Let’s call this the emotion video. Unlike many emotional intelligence tests 
in which the criterion for the correct answer is arguable (it could be expert opinion 
or subject consensus or muscle coding), the deception detection task has an 
unassailable accuracy criterion. What film the women are watching is a matter of 
fact. 

Ekman, Friesen and I administered the emotion video to many law 
enforcement and other professional groups as well as hundreds of college 
students. We were quite surprised to find that every group we tested obtained 
average scores that were no better than chance; that is with 2 choices, is this 
person lying or is this person telling the truth, the odds that you are going to get 
one right is 50%, and 50% is what we found was the average score in every 
group that we tested. This finding of chance accuracy was found not only in our 
studies but those of other deception researchers as well. They reported the same 
thing. And how often do you find everybody finding the same thing? So another 
one of our knowns is that most people cannot tell when other people are lying, 
beyond what would be expected by chance. 


148 



Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Volume 2, Part 2: Related Constructs 


Definitions 


• Social intelligence: .. .the ability to understand and manage men 
and women, boys and girls-to act wisely in human relations.” 

• Thorndike, 1920, p. 228 

• Behavioral cognition: “.. .the ability to understand the thoughts, 
feelings and intentions of other people insofar as they are 
manifested in discernible behavior.” 

• O’Sullivan & Guilford, 1975 p. 256 

• Emotional intelligence. .the ability to monitor one’s own and others’ 
feelings and emotions, to discriminate among them and to use this 
information to guide one’s thinking and actions” 

• Salovey & Mayer, 1990, p. 189 



Another finding from this research is that most people did not know how 
accurate they were in detecting deception. The correlation between their 
assessment of their skill at detecting deception, before they took the test, and 
their score on the test was zero. Other deception researchers have also reported 
the same thing, as has Patterson recently with respect to peoples assessment of 
their ability to judge their personality of others. So this is another known: most 
peoples’ opinions about their ability to judge others are unrelated to reality. This 
raises questions about the meaning of self report measures of emotional 
intelligence. If such scores correlate with successful functioning in life situations 
that is very useful, and self-reports can be used to make such predictions. 

And the question is why is this so? Are these differences due to 
motivational differences, differences in interests in thinking about emotions, some 
kind of social ability? This is another one of our unknowns. We do not know what 
those scores mean. So back to the situation in which we are testing all of these 
people and we are not finding any group where the average score is better than 
50% or chance. 
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Then in 1991 Ekman and I described a group of Secret Service agents 
who, as a group, did significantly better than chance on the emotion video. In 
1999, Ekman, Mark Frank and I described three other groups who were 
significantly above chance in their ability to detect deception. So we now have 
identified four groups of highly accurate lie detectors, highly emotionally 
intelligent people with respect to this particular task. We have found that although 
most groups do not perform well as groups within almost all groups there are one 
or two people who are highly accurate, but since these individuals occurred only 
about one percent of the time and had scored well on only a single deception 
detection measure, we did not pursue examining them. 

Then two things happened that made possible the study that I will 
describe. Mark Frank and Paul Ekman developed two additional deception 
detection videos and Ekman was invited to give a keynote address to 1200 
therapists. So rather than many small groups, we had one humongous group to 
test. 
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And now there were two additional lie detection measures. In one of them 
the crime video, young men lied or told the truth about whether they had stolen 
50 dollars. If they lied or told the truth successfully, that is, if they were believed, 
they received a financial reward. If they were not believed, whether they were 
lying or telling the truth they would not receive the reward, and they were told 
they would receive an additional punishment. 

In the second deception detection task, the opinion video, young men lied 
or told the truth about a strongly held personal belief, such as whether cold 
blooded murderers should be executed. As part of his keynote address to the 
1200 therapists, Ekman administered the 10 item opinion deception detection 
test video. He then gave the answers to the test to the therapists and asked 
those who had gotten scores of 90% or better who were interested in 
participating in a study of expert lie detectors, to write their names on the answer 
sheets. About 50 of the 1200 therapists received scores of 90% or better. By 
chance alone, 12 people should have scored so well, so we were hopeful that 
between twelve and fifty, there were some people who really were emotionally 
intelligent. Over the last eight years we have continued to lecture to various 
professional groups. As part of those presentations, Ekman, Frank and I have 
administered the opinion video tape. We asked people who received scores of 
90% or better to indicate whether they would be willing to participate in a project 
on expert lie detectors. We have tested many groups as well as individuals who 
contact us through media coverage of our work. 
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On the screen is the original criteria that we used to qualify people for 
inclusion in what we are currently calling “The Wizards Project”. After receiving 
scores of 90% on the opinion video and agreeing to participate, subjects are then 
given two other deception videos, the crime video and the emotion video that I 
described earlier. If they achieve scores of 80% on both of them, they are 
classified as wizards. We were calling them geniuses earlier but they are not 
geniuses in the ordinary academic sense. We started identifying wizards with our 
large group of therapists, but only 2 of the 50 that got scores of 90 % on the first 
test, qualified using what I’m calling the ultimate criteria. Then we noticed that 
some of the therapists obtained scores of 80% on one of the deception tests, but 
not on the other one, and the task they failed was almost always the crime video. 
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• 90% on one test 

• 80% or better on two additional tests 

• by chance alone: 

•25 in 1,000,000 



By this time we also had identified several potential law enforcement 
wizards: cops, judges, lawyers and arbitrators. And we noticed that they showed 
the opposite pattern, that is, if they achieved a score of 80% on only one of the 
two tests, it was always on the crime video that they succeeded and never on the 
emotion video. 
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So what you have here is a graphic illustration of what I have just told you. 
The law enforcement people did less well on the emotion video than the 
therapists. Almost all of the therapists did very well on the emotion video; only 
one of our 9 therapists did not do well on emotion, Conversely, the law 
enforcement people always scored 80% or above on the crime video but none of 
them scored 80% or above on the emotion video. 
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So we did a Chi-square analysis of these data, which was highly 
significant. So it seemed to us that what we were seeing here is the result of 
relevant learning experience, an illustration of tacit knowledge. That the 
therapists were good at emotion and good at opinion, which is what their 
profession entails, and not good at crime, makes sense. Recall, however, that we 
do have 13 ultimate experts, so we have 14 people who scored above 80% on all 
three tests. So we are going to be looking to see what kind of differences we can 
see, if any, among them. 
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This study is still underway. We’ve only interviewed about half of the 
experts so my report to you today is merely suggestive. We will be using the 
spouses or other close family member of each expert as their own control and we 
haven’t done any statistical analyses on these data. But I will report the result of 
our interviews with some of these experts, in which we interviewed them about 
their lie detection skills, using a think aloud procedure. 
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I was thinking as I was listening to the two psychometric talks that 
preceded me that I’m a psychometrician and I’m going to be drummed out of the 
corp because what I’m going to be reporting is essential a clinical study at this 
point; we call it an ideographic study to make it sound more scientific. 

What we do after the experts have been qualified, is that we play the 
videos and ask them to think aloud about what they are thinking, seeing, feeling, 
what is going through their mind, what are they paying attention to. I tape record 
this interview. This process takes from 2 to 4 hours. After that we give them a 
semi-standardized interview about their life experiences, their education, 
hobbies, work and all that kind of thing. 

I also found with some of the groups, for example, the cops that they 
would not be very forth coming about their exploits, They seemed very 
uncomfortable being self reflective and were inhibited in talking about 
themselves. With them I found that if I got a group of these guys together or get 
them together with their friends, have some drinks in a bar, and just kind of talk, 
they would be much more open and I would get many more stories that would be 
more revealing of their personal characteristics. I also have tried with some of our 
other experts to socialize with them, to have them and their spouse to dinner, to 
have people out to a restaurant, so I can get to have a more of a feeling of them 
as a person. 
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We will, however, be giving them personality tests of various sorts. I have 
delayed doing that because I want to get the perfect subset and as we heard 
from Gerard’s talk, it is very difficult to know exactly what the perfect measure is, 
but we will be doing that. Now these 29 wizards have been culled from tests of 
over 12,000 people, so they are an extremely rare group of people. We are 
conceptualizing this lie detection ability as an essentially normal distribution. And 
we are looking at people at the far upper end of the distribution. So what have we 
observed about the people that we have been studying? 
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• Baseline accuracy, both behavior and 
personality 



The first thing that we observed in the talk aloud procedure is that all of 
them are extremely sensitive to non verbal behaviors. They all talk about that. 
However, none of them uses the exact same non verbal behavior; some of them 
pay attention to facial expressions, but only the eyes; others pay attention to 
vocal intonations; other pay attention to how the liars and truthtellers move their 
heads. And they come up with and mention behaviors that we have not studied in 
30 years of looking at non verbal behavior. 

They are also paying attention to discrepancies. They will say “Well, his 
face looked like he’s relaxed but his voice got very soft” (that soft voice that Paul 
talked about earlier). In order to be sensitive to and to process these kinds of 
discrepancies, these individuals have to have a very great tolerance for 
ambiguity. They will notice these things and it doesn’t bother them that it doesn’t 
fit together. The few spouses that I’ve talked with will almost consciously correct 
ambiguities, saying “Well he does such and such, but that can’t be right, so I’m 
going to go back to the first thing.” 
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In order to tolerate ambiguity and come up with the right answer, many of 
these individuals take a long time to reach their decision. I was quite interested in 
the fact that all of them had already taken the test before I sat down to interview 
them, None the less they wanted to go through and pay attention to each item 
before they would say what it was they were thinking. And most of them were 
markedly different (from their spouses) in their ability to remember what they had 
thought and to give very detailed explanations of them. 

These individuals are extraordinarily highly motivated to do well at this 
task. They care about this task. Think about how we got identified “the wizards.” 
We found them because they were attending a workshop on interrogation, 
improving your interrogation skills, or a workshop on learning how to detect 
deception. And we said “Okay you did very well. Are you willing to participate in a 
study/” Then they had to send me a consent form. They had to take two more 
tests and return the video tapes to me. They had to agree to be interviewed. So 
they were motivated to learn as much as they can about this. Another example of 
the degree of their motivation in this area: one of our early wizards calls me at 10 
o’clock at night and he says: ’There’s an American Justice show on TV and it just 
started (it was like 1 or 2 minutes into the show) I think that man is telling the 
truth, and I want you to know that before the show is on that I think he is telling 
the truth and if I’m wrong I want to call you back and talk to you about why I was 
wrong.” This is 10 o’clock at night, This is a man who has been retired from his 
law enforcement career for 5 years. So they are tremendously motivated to get it 
right. 
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Paul talked earlier about how difficult it would be, to widen the gap 
between the spark and the flame, the impulse and the action, and that it takes 
habitual practice even for a man like the Dalai Lama. Paul reported that he can’t 
go 3 days without losing that ability. These individuals, and there is a range 
among them, only a few are fanatically driven, and are doing this habitual 
practice all the time. But all of them care. It’s a part of who they are and they are 
interested in it, and they practice it. 

The last thing is a bigger category. I have been quite impressed with these 
individuals with how they construe and understand the personality of other 
people. Yesterday, Karl Heider said that the Minangkabau can respond in terms 
of our categories for emotion, but that’s not the way they ordinarily do, and this 
was implied in Gerard’s talk as well, that if you use indigenous language 
structure you may get a different factor. And what I’m seeing with these expert lie 
detectors is that they understand discrepant non verbal and verbal behavior in 
the light of their understanding of who that person is but they use personality 
templates that are very different from our extroversion, introversion, 
conscientious, etc. They don’t talk about those kinds of traits. It’s not that they 
couldn’t, but they have what seems to me fairly idiosyncratic ways of 
understanding other people. 
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One man, for example, said “He looks like a choir boy who hasn’t been 
molested”. What that implies, you see, is that he has a template of what a choir 
boy who has been molested is. And what that also implies is that he has seen 
such people. Another man described an African American man with Rastafarian 
hair curls and a nose ring, as looking and acting like a second grade girl. And 
emotionally, that’s who the guy was. If you paid attention to the Rastafarian hair 
and the nose ring, you would be totally off base. But this astute man could see 
beyond the hair and the nose ring to get a sense of the emotionality of this 
person. Using that as his baseline, he was able to accurately assess deviation 
from that baseline in behavior and non verbal discrepancies and so forth. I am 
calling this the Miss Marple effect, from the Agatha Christie character who solves 
crimes by comparing the personalities of the people involved in a crime she is 
dealing with to the personalities of the people from her home town. And obviously 
the way in which she construes those personality characteristics are very 
different than we psychologists would. 
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So far I have reviewed ways in which the wizards are similar to one 
another. This slide shows some of the ways in which they may differ among 
themselves and from other people. After we do the statistical analyses, we may 
find that more of the wizards than the control group have verbal sensitivity, but at 
this point what I’m noticing is that among the wizards they all pay attention to non 
verbal sensitivity but they differ in terms of how much emphasis they give to the 
nuances of language. A number of our experts are lawyers and they were 
particularly astute in terms of changes in the level of words that people would 
use, whether they would speaking at a very highly educated, articulate level and 
all of a sudden use a very inappropriate word, or would try to use a very fancy 
word when their ordinary level of speech was not at that level. But not all of the 
wizards do that. 

The first variable on the list is gender. The non verbal sensitivity literature 
says that women are more sensitive than men in the ordinary kind of lab work 
that we do. In the expert lie detection work, however we are finding that it is really 
unpredictable at this point. We find more male therapists than you would expect 
by chance, given the sample we had; we are finding more female cops than you 
would expect by chance, given the samples we had. Gender seems not to be 
predictive of the extremely high level of accuracy in lie detection that we are 
studying. 
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The issue of childhood trauma is also still up in the air. Among our early 
wizards, many of them reported abusive childhood situations, parents who were 
either full of rage or alcohol, alcoholics or they beat them, so they reported a life 
time of having this kind of emotional scanning going on, where they were 
constantly aware of shifts in the emotionality of the people around them. But 
more recently our other experts are not reporting that. Now I am still continuing to 
probe for instances of early childhood trauma, but so far I’m getting mother and 
apple pie and everything was just fine and they were just people who were very 
curious and this was just one of many areas about which they were very curious. 

In terms of education everybody in our sample has at least a college 
degree, because many of the cops, even if they only had high school degrees 
when they entered the police corps, usually would get further educations later. In 
terms of intelligence, there is a range. I think there are some that are just a little 
over a hundred IQ and there are some who are PhD, MD, PhD/JD’s, extremely, 
extremely brilliant. So in terms of verbal intelligence there is a range. 
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The professions are variable. We have the large group of therapists I 
described earlier. In the final sample, we have 5 therapists out of the 29 wizards. 
We have a number of law enforcement personnel, local cops and federal agents. 
We have many arbitrators or labor lawyers and dispute mediators. Those were 
the groups that gave us the largest yield. So people who are in a situation where 
they don’t have a side, where they are trying to make a decision between the 
employer and the worker and if they don’t get a reputation for being even handed 
and fair they are not going to continue to be able to work. So that group had a 
very high yield rate. Dispute mediators also have a very high yield rate. Other 
differences may include sensory acuity. Many of the experts of the wizards report 
stories or others would tell stories about their ability to scan an environment, not 
only for social or emotional information; their scanning was for all kinds of visual 
sensory details; one of them is a bird watcher, another one has expertise as a 
hunter and is well known for his ability to trap in nature. So this kind of sensory 
acuity is something we will be looking at. 

The other thing that some, although not all of the wizards report, is role 
playing ability. Wizards who seem fairly quiet and unobtrusive will tell stories or 
stories will be told about them of their working undercover, of their assuming 
characteristics totally unlike their everyday life, so many of them have this 
capacity. But not all of them do. 
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Another difference, that is not on here, but is in terms of the personality of 
these expert lie detectors. Many of them are introverts, and you would suspect 
that perhaps using our procedure that there might be a bias for that, since how 
you get qualified is that you are willing to watch a video tape and make an 
assessment of somebody else doing an interview of somebody. However there 
are 2 or 3 of the wizards who are flaming extroverts. They are not the quiet shy, 
observant types that some of the others are, but alpha personalities. Also some 
of these people who are very quiet may not be introverts. One wizard said to me 
“I’m quiet, but I’m not shy.”, so he lays back and when he’s ready and he is able 
to do the work, then he does it. Many of the wizards are over 40, but we have 
one law student who is 25. Most of the other wizards, I think, have this 
tremendous understanding of people’s personality because their lives have been 
extraordinarily broad, deep and rich. One man is an arbitrator, has his blue suit 
on, but over the course of his life he has worked in the Appalachians with wild cat 
coal mine operators, he’s worked on tug boats on the great lakes, he’s worked 
with the lowest level of workers and now he is working with CEO’s of 
multinational corporations. So he’s just had a life experience, for example, that I 
have not had. He’s seen and worked with all kinds of people, and obviously many 
of the law enforcement people, also have this broad array of experience with 
people. 
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And that’s why I think emotional intelligence does not exist in a vacuum. 
Moshe talked about display rules, and Paul talked about subtle expressions and 
leakage, micro momentary, so it’s not enough to really recognize full facial 
expressions. That’s easy. But when people moderate those expressions, for 
cultural or personal reasons like trying to appease somebody, that’s the subtlety 
that, I believe, reveals emotional intelligence. And you need many kinds of skills, 
motivations, and personal attributes in order to do that. To my mind, it’s like 
training Olympic athletes, you need to have basic talent, you need to get 
feedback, you have to have motivation on the part of the person that wants to 
become the Olympic athlete, and that potential athlete has to practice, practice, 
practice and that is essentially these what wizards of lie deception do. Thank you 
very much. 
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DISCUSSION 

DR. KYLLONEN: Questions? 

DR. GADE: What about actors? 

DR. O’SULLIVAN: Actors I think will vary, I think they are in a business 
presenting a performance and we have not found any actors who are particularly 
good. We have one artist, who did painting of facial expression of emotion, but I 
think you have to be interested in what other people are feeling, not in what you, 
yourself are feeling, so I wouldn’t expect a high yield rate from actors. 

DR. STRICKER: Maureen, after you mentioned that this secret service 
group did above chance, you said there were three other groups. What were 
these three other groups? 
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DR. O’SULLIVAN: The other groups were selected federal investigators, 
if we just test the CIA forget it, they’re 50%, FBI 50%, but there was a workshop 
where people in each of these agencies, FBI, CIA, and all that, were asked to 
send their best interrogators to an advanced interrogation course and that’s who 
they were, they were people who in their agencies, people knew were good, at 
understanding others, the other group was a group of forensic psychologists who 
took three days to study this testing deception, so they gave up private practice 
days, and they gave up their weekends to come to this three day course so 
there’s that high motivation which we referred to and the other is a group of 
judges, again who was interested to take the course, so that’s who they were. 
None of those people are in this group, because we didn’t collect this kind of data 
then. 


DR. LEN WHITE: I recall reading a study a few years ago, where customs 
officials blew away all the other professionals. I’m thinking that in terms of related 
to an implication that people and professionals where they have opportunities to 
observe people who might be deceiving and also get feedback on whether they 
were right or not, might be an important thing. Now have you considered asking 
people things like how many people to you see a day, where you are trying to 
decide whether they’re deceiving or not and do you get feed back on that? 
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DR. O’SULLIVAN: I think you are absolutely right. In fact Paul’s article 
was about and I have an article out 2003 and PSPB and I have something like 17 
reasons why we can’t detect deception. But I think that one of the major ones is 
the fact that we don’t get good feedback like we’ll say I can always tell when my 
son is lying. But of course we only know the lies we caught him in. You know, we 
don’t know what a good lie looks like, we don’t what a good lie looks like, we only 
know what a bad like looks like. So I think the lack of feedback is a very 
important thing, and the other thing about the custom agents is that they don’t 
have a side in a way, you know like a copy doesn’t arrest somebody to be mean, 
they arrest them because they believe they believe they are guilty, and so that 
belief will color a whole bunch of things that they will see. And I think that is why 
the arbitrators and the dispute mediators much like the customs officials they 
may have a more balanced and open view about it. 

DR. SIGEL: Do you think that there is any thing in terms of the flexibility 
these people have to make these judgments and whether they are open to begin 
with? 
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DR. O’SULLIVAN: Yes absolutely, and that would be a part of the fact 
that these broad and rich lives that they’ve had. They have sought that kind of 
diversity in their life, and they will look at somebody, like there is one guy 
everybody gets wrong on the opinion video cause he looks like a total goofball. 
And people will look, they will make this instantaneous attributional judgment and 
as we know, once you’ve made those judgments, it’s almost impossible to 
change it, but these people don’t make that judgment, they delay, and they say, 
well he looks a little goofy, what’s he’s doing in this study, okay and they keep it 
in mind. And what’s interesting is these people like in terms of isms they are in 
their personal lives, many of them are extraordinarily bigoted and they will say 
outrageous things, that no academic would ever hear said. But when it comes to 
judging an individual person that’s out the door, they pay no attention to what 
they’re, so they are able to decouple and to be extraordinarily flexible in that way. 

DR. SIGEL: But, they are also able to delay being reflective. Is that what 
you’re saying? 
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DR. O’SULLIVAN: Yes, and I have one woman, this artist who I could see 
learning as she was doing the test. Because she noticed a particularly kind of 
behavior with the first man who, on the basis of other evidence she thought was 
telling the truth and then she saw the same thing, later on in about a man about 
who she was more equivocal and so she was essentially teaching herself as she 
was going through the tape. And that is tacit knowledge ability that I think is so 
interesting; where as, I’ve only done a couple of the spouses and they are much 
more wanting to get closure, wanting to make the decision of a lie or telling the 
truth. Where as the attitude of these experts are what is this wonderful thing I 
have before me? From what garden does this flower grow? 

PARTICIPANT: I was interested in your comment about being bigoted, 
could you elaborate a little bit, because I have a follow up question. 

DR. O’SULLIVAN: We have the N word, how women are so stupid, and 
I’m right there and they tell me that. 

PARTICIPANT: They want to categorize, I was interested in taxonomy 
because you said the bird watching and some other things you said, intrigued me 
about the need to have a taxonomy of these various categories of nature, or 
other stimuli that they use the taxonomy to help understand their environment. 
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DR. O’SULLIVAN: I think the bird watcher was more about his interest in 
subtleties of sensory experience and he bird watched more for the pleasure of 
differences among them, and that kind of attitude of the pleasure of differences 
among them is more like what I would see. I don’t see them looking to pigeon 
hole people, I think that’s what we psychologist’s do and I think that’s why 
academic psychologists as a group gets 50% on these tests, you see, but when 
you are looking to find out how the people are different then you have a different 
mindset. Does that answer your question, we can talk later. 

DR. KYLLONEN: I have a question while we’re waiting for the next 
presentation to set up. Could you say a little bit more about the validation of 
these ideographic observations or findings? 

DR. O’SULLIVAN: At this point it’s very low; it’s just how sensitive and 
emotionally intelligent you think I am. But what we are planning to do, however, is 
to use their spouses as controls, so we will content analyze the typed script of 
these videotaped interviews and do content analysis comparisons between the 
experts and the control. I also will be giving both the controls and these experts a 
variety of tasks on intelligence and what not. At this point, it’s just to share with 
people an element of what an emotionally intelligent person might look like from 
a wide variety of professions. And what is so interesting is that these are lie 
detection measures and yet we’re getting people who are from a whole variety of 
different fields and that’s why we think there is some cross generality in the in the 
observations. 
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ATTENTIVENESS AND POLITENESS IN LEARNER-COMPUTER 
COMMUNICATION 

DR. KYLLONEN: I would like to introduce our final speaker this morning, 
Professor Lewis Johnson from University of Southern California, Information 
Sciences Institute. 

DR. JOHNSON: I’m from a computer science background but I have been 
raiding the emotional intelligence literature quite a bit, as well as some of the 
other related disciplines. What I’m going to be talking about here is some of our 
work in modeling, attentiveness, and politeness in learner-computer 
communication. 
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The context of this is to create animated virtual tutors (we sometimes call 
them guidebots), that appear on a screen or in a virtual environment that are able 
to engage with learners in natural interactions to promote learning. We use these 
as guides, tutors, teammates, or as other story characters. This is relevant for 
education and training but this kind of technology is also applicable to commerce 
and entertainment applications. As an aside, I just want to say that in order to 
make these agents or guidebots work effectively, they have to have good 
emotional expression and often have to incorporate rich models of emotional 
processes. A lot of our work where this comes up is in what we call Interactive 
Pedagogical Dramas, which are dramatized scenarios where learners can 
understand how skills can be applied. 
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This shot here is from a project that we did called Carmen’s Bright IDEAS. 
Stacy Marcella, Kate LeBoran and I fielded this at a number of cancer clinics 
around the United States. What it is intended to do is to help mothers of pediatric 
cancer patients learn to cope with their problems, many of which have a lot of 
emotional impact. There are a couple of aspects to this, one of which is to be 
able to create fictional characters that present the problem solving activity. In this 
case Carmen, the woman on the right, is a mother of a child with cancer. We did 
a lot of work looking at the emotion and the clinical psychology literature to create 
realistic expressions of emotion. 
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And then we created a virtual counselor, Gina, who responds to what 
Carmen says and does. Carmen, at any one time, can respond to Gina or to the 
learner who can influence what Carmen’s emotional state is in the situation and 
to how she is coping with the situation. And to make this work effectively, this 
requires rich models of emotion including appraisal and coping which is realized 
in the Carmen character in particular. That is not what I’m going to be talking 
about today; I just sort of put that out as a teaser. 
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What I’m going to be talking about today is what technology is required to 
create more of these virtual Gina’s, but which are able to interact with human 
learners in an effective way and which can operate in a range of learning 
settings, including more ordinary learning domains, such as math and science. 
Although we certainly recognize that emotional intelligence is important, the thing 
that we found to be of foremost importance is developing a model of social 
intelligence that we can incorporate into these agents. Primarily, the reason for 
that is to influence the motivation of the learners who are interacting with these 
guidebots. As I will explain, what we found is that motivational factors are things 
which good human tutors are very attentive to and we want to create virtual 
tutors that have similar capabilities. We refer to this as social intelligence 
because we regard the influences on the learner as taking place in a social 
context. Like Rosalind Picard, we are also influenced by the research of Reeds 
and Naps and others who liken human-computer communication to human- 
human communication. So, we actually found it useful to emulate aspects of 
social interaction in the agents as they interact with learners. What I am going to 
be talking about mainly today are two aspects of this that we have looked at in a 
fair bit of detail and think are important parts of this. One, which we call 
attentiveness, is the ability to observe closely what a learner is doing and detect 
potential problems before they become a serious motivational or emotional issue. 
The second is the role of politeness in interacting with the learner; basically how 
politeness can be exploited to have a motivational affect. 
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The context of this particular work that I’m going to be talking about is a 
project that we have doing for a while called the Virtual Factory Teaching 
System. This is an on-line simulation of factory processes and a number of 
universities have used this for teaching industrial engineering. It also is a topic for 
beginners who might potentially benefit from this, such as business majors, who 
find this difficult to master initially. Our goal is to be able to incorporate into the 
Virtual Factory Teaching System a guidebot that can monitor a learner’s activities 
and provide help and feedback in a socially appropriate manner. 
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Some initial experimental studies that we did form the basis of this work. 
We videotaped human tutors interacting with human learners in front of the 
Virtual Factory Teaching System. We scheduled multiple sessions so we could 
compare the interaction between the tutors and learners at later sessions as 
opposed to the initial sessions. The learners carried out a series of exercises and 
the tutors were instructed to provide them with assistance as they deemed it 
important. 
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One of the conclusions that we drew from this was that there was a wide 
variation in preferred interaction styles. One difference that we noted among the 
learners that we studied is that the way that they interacted with the tutor seemed 
to relate to whether they preferred to work on problems in a collaborative manner 
or preferred working alone; this would clearly seem to relate to personality 
measures that were talked about earlier this morning. We also noticed that there 
was a wide variation in confidence between the learners as well as a difference 
in confidence over time and that the human tutors were able to assess the 
learners’ degree of confidence ability and preferred interaction style and then 
respond to that. So what we were trying to understand was how is the human 
tutor able to recognize these factors and then adapt accordingly? 
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We looked closely at videotapes of the interaction and then followed this 
up with interviews of the tutors and these are some of the factors that we found 
to be most important. First, are the tutors’ expectations derived from knowledge 
of the task. Basically, the tutor would look at what the student was doing or 
reading and from that, draw expectations of what the student ought to be doing 
next and if the student seemed to be at a point of indecision, the tutor might jump 
in. Focus of attention was extremely important which was inferred from eye gaze, 
where the student was looking on the screen or the mouse location. It was also 
clear that the tutor was trying to infer a degree of confidence; a lot of that was 
inferred from the kinds of questions and comments that the learners gave and 
the extent to which they seemed to be looking for confirmatory feedback from the 
tutor. Also, the amount of effort that was expended was important. As the student 
would go through these exercises the tutor would draw a general estimate of how 
heavily engaged the learner was from this pattern of interaction. So, these are 
some of the things we wanted to focus on and what we wanted to see was would 
it be possible for us to then create a guidebot that could draw similar inferences 
from similar sets of information. You see in some ways that these are similar 
objectives to some of the stuff that Rosalind Picard described, but we have 
chosen not to use intrusive measures such as galvanometers and things of that 
nature. We wanted to see how far we could get with the kind of cues that human 
tutors rely on, more than what a conventional desktop like what I am standing in 
front of here is able to respond to. 
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One of the key differences we wanted to achieve in terms of behavior was 
to create an agent that can proactively offer help. Again, we saw in Rosalind’s 
talk the discussion of what some of the problems are when an agent jumps in 
with advice at an inappropriate time. How can we avoid that? That is clearly a 
dangerous thing to do if the agent has a poor understanding of what it is that the 
student is doing and why, so we need to be able to do a better job there. In order 
to support that, we created an interface that the student interacts with - here are 
the major components of it. On the bottom right is a tutorial window where the 
learner can read both information about the topic and instructions about what to 
do. The top left is the interface to the Virtual Factory simulation and the top right 
is the window for communication with the guidebot. The BFTS window and what 
we call the web tutor window are both instrumented, so as the student is 
interacting - scrolling, pressing buttons etc., all that information is being captured 
so it can be analyzed. 
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An important additional capability that we have been exploring is the use 
of gaze tracking to monitor what the learner is doing. Our goal here is to be able 
to do this in a robust way, without calibration and without special head 
stabilization hardware or anything similar to that; we want to make the effective 
use of gaze as one of the input modalities that the system can use. This 
particular system is based upon work by Larry Kite and his colleagues in the 
Laboratory for Computational Biological Vision at USC. Basically, it utilizes a low 
cost camera perched on the top of the display, which is focused on the learner’s 
face and is looking for particular landmarks around the eyes, pupils, and the nose 
and then uses that to estimate head pose and direction of eye gaze. 
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Inferring Learner Focus 



That works reasonably well, although without pre-calibration it’s prone to 
error so we’ve been combining that with other sources of information that we 
gathered from the interface, as I mentioned, actions such as scrolling a window, 
clicking a mouse, typing, etc. These are all combined with the information from 
the eye tracking system in a dynamic Bayesian network. So when the student 
clicks on an object, we assume with high likelihood that the student is focusing 
on that object at that point in time. As time progresses, the likelihood that the 
student is still focusing on that object will decrease, but when we combine that 
with the information from the eye tracking system, that improves the overall 
accuracy of interaction. And what we’ve observed in our experiments is it actually 
does a pretty good job with the one proviso that there is a slight delay in 
recognizing when the student has focused from one new area of the screen to 
the other. But, it is adequate for what we are trying to do here, which is to model 
what we saw human tutors doing which was to be able to see what a learner’s 
primary focus of activity is. 
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However, that by itself it is not enough. We need to combine that with 
some understanding of what the learner is doing in order to form expectations of 
what the learner is likely to be able to do next. What we combine that with is a 
plan recognition system which is basically taking this data from the user 
interaction and comparing that against patterns of different types of task 
solutions, which uses that to follow learner progress. Associated with those plans 
is not just the steps taken to perform but also the amount of time that we 
estimate is required to perform that task. We also incorporate into this estimation 
how much time it’s going to take to read the instructions within the tutorial so that 
the system can draw expectations of when the student is likely to engage in 
actually carrying out the next step in the task. Given this time information, we can 
detect periods of indecision as well as incorrect actions by themselves. 
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We also coupled this with a way that the learner can provide self-report of 
their own motivational state at various points in time. Colleagues at the University 
of Edinburgh have experimented with these types of interfaces and they found 
that they are reasonably successful. The factors that we are particularly 
concerned with here are: self confidence, sense of self control and sense of 
mastery of the subject matter area. Particularly, confidence and self control are 
the things that are the most likely to vary over the course of a session or a series 
of sessions and so these are ones that we want to try to track. So, we make use 
of the self-report interface but we want to make use of these other sources of 
information so that we don’t have to rely upon self-report entirely. 
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We are currently working on this and we see a number of things that we 
can derive from the data that we are collecting automatically. One is effort, which 
is an overall estimation of how much time the learner seems to be performing on 
reading the material and carrying out the task. Does it appear that the student is 
going though the material thoroughly or just glancing through it? We can detect 
instances of indecision at points where we expect that the student should be 
performing an action and we can tell that they are attending to the screen but 
they’re not performing the action; there must be something that is causing them 
not to go forward. And then also, we see frustration as something that can 
potentially be derivable here by looking at instances of actions which fail to 
indicate progress and to suggestions where the student seems to be having 
difficulty accomplishing their goals. Finally, another important issue is to 
understand what the overall pattern or mode of work the learner is engaged in. 
That is, is the learner systematically going through and solving a problem, or is 
the learner just exploring the interface and the material? We don’t want our 
guidebot to jump in with advice about what to do if the student is not really 
currently engaged on the task. 
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Again, this is something than can be derived by looking at the patterns of 
activity; that when we see that the learner is going through this process of 
reading material and carrying out actions and then reading some more, we can 
then provide feedback on that basis. But, if we see that they are carrying, for 
example, a series of actions on the simulation interface without reference to the 
tutorial materials, or vice-versa, those are indicators that a different mode of 
activity is taking place. So, those are the mechanisms that we are employing in 
order to be able to attend more closely to the learners’ activities. It doesn’t utilize 
the kind of sensors that Rosalind was talking about, but we think it is suitable for 
detecting a wide range of intervention points. 
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Then the question is how to intervene when one of these intervention 
points has surfaced? For this, we have been trying to develop a model what we 
call Interaction Tactics, which is basically to determine how the system should 
intervene at a given point. We want to examine how to help the learner in a way 
that respects the learners’ autonomy and sense of control, particularly if we know 
that there is some deficiency in the learners’ sense of control. We also want to 
examine how to influence the learner to promote learner motivation in a manner 
which is sensitive to the social relationship between the guidebot and the learner. 
We are proceeding here with a hypothesis that such a relationship exists and it 
should serve as the basis for governing the interaction. This is something we are 
in the process of testing. 
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Examples 


Tutor phrased comments in order to 
reinforce learner control and joint 
activity. E.g.9 

7i “Why don’t you go ahead and read your tutorial 
factory” 

7i “You want to save the factory” 

7i “I’d skip this paragraph” 

71 “So why don’t we do that?” 

Tutor avoided giving direct 
instructions 

7i Except for operating the interface 
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So, what do I mean by the different ways in which a tutor can influence the 
learner? Well, they contrast what I’m about to describe here. The typical 
feedback that you get from computer-based learning where you perform some 
action and it tells you, “No, that’s wrong, try again” or “That’s wrong here’s the 
right answer”, we saw in our videotapes very few instances of human tutors 
telling the student, “No you are wrong, do this.” Instead, we saw a lot of 
phraseology like the following: “Why don’t you go ahead and read your tutorial 
factory”, or “You want to save the factory”, or “I’d skip this paragraph if I were 
you”, or “So why don’t we do that.” So these are all trying to convey suggestions 
or advice but they are not direct instructions of what to do. The only place where 
we found direct instructions were places like in operating the interface which 
said, “Click here in order to move forward.” We wanted to try to develop a model 
that would enable our system to generate similar advice in a similar manner for 
similar reasons and, of course, understand what the real reasons are underlying 
this. 
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Theoretical Framework: 
Politeness (Brown & Levinson) 

■ Social actors motivated by face wants 

7i Negative face: freedom of action and freedom from 
imposition; autonomy 

7i Positive face: consistent self-image, and desire that self- 
image is appreciated and approved of by others 

Face-threatening acts pervasive in interaction 

7i Warnings, offers, promises, challenges, emotional 
displays 




Face threat depends upon power, distance, 
ranking of threats due to social context 

Social actors employ politeness tactics to 
mitigate face threat 
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To develop this framework, we drew on the work of Brown and Levinson, 
particularly their model of politeness, where they looked at how polite 
communication takes place in a range of different cultures. Their model basically 
is as follows. Social actors, in general, are motivated by face wants, both 
negative face, that is the desire for autonomy and freedom of action and positive 
face, which is a positive self-image and a desire that that self-image is 
appreciated and approved of by others. In the course of interaction there are 
various kinds of interactions which can potentially threaten positive face or 
negative face or both. Warnings, offers, promises, challenges, emotional 
displays- these all potentially can be face-threatening acts and therefore people 
employ politeness tactics in order to mitigate or even avoid those face-threats. 
The extent to which those face-threat mitigation strategies are applied depends 
upon on the relative power relationship between the parties, the social distance, 
as well as the ranking of threats due to social contexts or culture. 
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Role of Politeness Factors in 
Tutorial Interaction 

■ Common tutorial actions (criticism, advice, 
hints) are face-threatening acts 

Tactic failures impact agent’s positive face 
Face threat depends upon distance 

7i Distance depends on duration of interchange, 
established trust, learner’s negative face wants 
(preference for autonomy vs. collaboration) 

Choose tactics to promote learner positive 
face, mitigate negative face threat 

7i By promoting shared goals 

7i By avoiding direct instructions 

7i By reinforcing positive (self-)assessment of goal 
achievement 

7i When dictated by social distance, learner motivational 
factors 
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What we found is that we could apply this notion to understanding the type 
of interactions that take place in a tutorial or coaching setting. First of all, 
recognize that criticism, advice and hints, are all potentially face- threatening 
acts. They threaten positive face by implying that the learner is not doing a good 
job at what they’re doing and they are potentially threatening negative face by 
controlling what the learner does and giving them less autonomy. We also noted 
that it’s possible that if a tactic fails, that would impact a speaker’s positive face 
and this seemed to us to be something that is potentially relevant for agents that 
make mistakes and need to apologize to the user when that mistake occurs. We 
asked the question of whether the face-threat depends upon distance and we are 
still analyzing our human-human transcripts, but it does appear as if there is a 
tendency there, at least for interacting with some learners. That is to say, we 
noted that in second sessions of our studies, that there were many more 
instances of the tutor saying, “Okay no, do this now”, much more of what Brown 
and Levinson would call bald-on-record interventions. This suggested that there 
is some change in the interaction over time which made the tutor be more 
comfortable in being more direct in their criticism. Also, we thought we could 
potentially build on a tactic that Brown and Levinson talk about, which is to 
promote positive face which then promotes the sharing of goals between the 
speaker and the hearer. This is what teachers have to do a lot of; they need to try 
to get the learners engaged in the activity that they are involved in and to share 
the goals of the teacher. So the question is, to what extent could the politeness 
model actually be a method for trying to promote that? I think to some extent it 
can. 
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Example Interaction Tactics 

■ Rhetorical requests to give hints 

“Can I give you a hint? Try this..,” 

7i Question reinforces learner negative face; failure to 
wait for answer avoids positive face threat 

■ Hints phrased as questions 

7i “Do you want to do x?” 

71 Reinforces learner control (positive face), can 
influence learner goals (positive face) 

HHints as suggestions 

7i “You could do x.” 

7i Similar face effects as questions 
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To give you a sense of how we apply the Brown and Levinson model in 
particular instances, I’ll analyze a few of the tactics that we observed in our data. 
Here, for example, we observed a tutor saying, “Can I give you a hint? Try this”. 
So, phrasing it as a question reinforces learner negative face because it’s then 
up to the learner to either accept that hint or not. But notice that offering a hint is 
potentially face-threatening because then the other person has to decide whether 
or not to receive it. So, by going on and saying, “Well, try this”, as we would 
analyze it, it’s a way of reinforcing negative face from the advice but avoiding the 
negative face-threat of the offer. When hints are phrased as questions, we do 
similar analyses in terms of negative face. In terms of this group of learners, most 
of what was going on was having the tutor reinforce negative face on the part of 
the student, but we did see that these could potentially have a role in influencing 
the learner goals. For example, do you want to do” X” now as sort of suggesting 
that the learner should adopt that goal. 
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Interaction Tactics (Cntd.) 



Hints as suggestions of joint goal 

7i “Let’s do x.” 

7i Suggestion mitigates negative face threat; 
reference to joint goal influences positive face 
wants; depends on learner autonomy preferences 

Hints as references to tutorial authors 

71 Deflect blame for face threat to authors 

Imperative hints 

71 Used only when blame is deflected (l.e., to 
interface), or possibly when distance is reduced 
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In a more extreme case, like the suggestion of a joint goal, “Let’s do this 
now”, this is something which is addressing both negative face and positive face. 
This is one of the standard tactics that Brown and Levinson refer to, that one way 
of promoting positive face is to promote joint activity. The places where you 
would expect that there not to be these kinds of face-threatening acts are the 
places where there is no threat of a face-threat being made by the tutor. In fact, 
what we see is consistent with that, that what the tutors we studied did a lot of 
was say, “Okay, the tutorial says you need to do this now.” Since the blame is on 
the tutorial, then the tutor can be more direct in terms of what it can say. 
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We now realize this in the form of a complete social intelligence system, 
which, on the left, has these abilities to track the learner’s state, both at a 
cognitive and motivational level and also to detect interaction opportunities and 
report changes in a learner’s state. Then, we have developed a tactic selection 
interface which selects an appropriate tactic. This can be used in two ways - we 
can use it under control of a human experimenter as well as under automated 
control or some combination of the two. This gives us some freedom in terms of 
what experimental approaches we want to take. The politeness module decides 
how to realize a particular tactic in English and is ultimately coupled with the 
appropriate non-verbal gestures. Finally, the natural language generation and the 
gesture commands go back to the guidebot persona to determine how it then 
interacts with the learner. 
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. 


Choosing Politeness Level_ 

■ Choice of politeness strategy depends 
on: 

Inherent threat of action against learner’s positive & 
negative face 

7i Social distance between guidebot and learner 
7i Desired influence on learner’s positive face 
7i Desired influence on learner’s negative face 

Tactic generator chooses from a library 
of utterance patterns, each achieving a 
different amount of face redress 
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How do we implement the politeness model in this fashion? Actually, we 
had to extend the Brown and Levinson politeness model to some extent. We 
found that in order to be able to choose the appropriate strategy, we needed to 
take different categories of tactics and classify them both according to their 
potential negative face-threat and their potential positive face-threat and then 
choose the one which has the appropriate evaluation for each of these, 
according to the particular type of face-threatening action. For each type of face- 
threatening action, advice, etc., we would collect a set of tactics that could be 
performed and then the politeness model chooses the right one from among 
those. Now, this can be varied by the social distance that is specified. In other 
words, there is a parameter to this model that you can specify, which is a slider 
that you can control a greater or lesser social distance. As you reduce social 
distance you get more instances of bald-on-record advice. Finally, we provide a 
way of directly augmenting the desired influence on learner positive face and 
learner negative face. Again, this is an extension of the Brown and Levinson 
model; we’re saying that we want this agent not to just be polite but by being 
polite, to reinforce learner autonomy or self-confidence. So, we realize that here 
by giving you the ability to augment the importance of negative face-threat or 
positive face-threat or both. Then, the tactic generator chooses from a library of 
utterance patterns that most closely matches the parameter setting. 
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Planned Evaluations 


Of attentiveness: 

7i That attentive guidebots are more effective in 
promoting learner progress than reactive ones 

Of politeness: 

7i The polite guidebots gain better user acceptance, 
and have a better impact on learner motivation, 
than rude ones 
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This is work that we are currently doing and we’ve got a process of setting 
up a couple of evaluations here. One question that we are interested in is how 
much additional advantage do we get from this kind of attentive learner modeling 
activity versus a more reactive one? This gives us the potential for a larger 
number of intervention opportunities. We need to evaluate that both from the 
learners’ perspective - were these appropriate interventions, and also compare 
this against what a human expert tutor would say. Again, we use this design in 
order to support that we can set this up so that the system is operating 
automatically but a tutor can be sitting in another room observing what 
recommendations the system is making in terms of intervention opportunities and 
commenting upon their appropriateness. As far as the politeness model goes, we 
certainly don’t claim that this is a complete account of how social intelligence is 
manifested in computer-learner interaction, but we want to understand this 
particular mechanism in detail, mainly what effect that it could have. The way that 
we see to do this is to compare the version which I described, which is trying to 
deal with these face-threat issues versus a condition which ignores these, which 
freely criticizes the learner whenever they make a mistake and makes direct 
instructions of what to do. And then we would evaluate that, not just in terms of 
the learner performance and the learner attitude toward the system, but we also 
we want to understand what the impact on motivational state is. Are we right that 
this approach will actually result, after repeated interactions, in giving learners a 
greater sense of self-efficacy in their interaction? So I will stop there and 
entertain any questions. 
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DISCUSSION 

DR. KYLLONEN: Questions? 

MS. PLANTE: The reference you made to the difficulty and mastery of the 
system and its use currently in higher education programs. What are your plans 
to look at that from other perspectives and simplifying the systems such that they 
could be used in support of K12 learning? 

DR. JOHNSON: I think that there are always things that could be done to 
improve the system but, I think our particular interest here is in understanding 
where individualized feedback and assistance would be advisable. I guess what I 
would answer is that we could certainly do this but the point of this is not to 
improve this particular system per say but to develop a set of techniques that 
could be applicable to a range of different learning applications in order to 
provide assistance to learners when they are getting started. We regard that as 
being a general problem that is worth trying to address. 

DR. KYLLONEN: You looked at what the human tutors did and you 
captured all these activities or tactics. My question is: Did you make any attempt 
to try to sort out which of those activities were more effective on the human 
tutor’s part? Or was the assumption that these were good tutors who employed 
effective tactics? 
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DR. JOHNSON: We did a lot of analysis of one tutor in particular who had 
received awards of excellence in teaching and so we presumed that these tactics 
would likely be a factor. But, we haven’t done the comparison that you describe 
of course we can now easily go back and do that, as I said let’s take out all of 
those tactics and then compare. Then, once we see the main effect, one of 
course can do further analyses and say what’s more important here, to address 
negative face or positive face, or are there some other things we should be 
taking into account instead? So, that would be a topic of further investigation. 

PARTICIPANT: How are you incorporating the gaze tracking and the 
other measures into the mentor’s interactions? 
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DR. JOHNSON: I neglected to mention before that I did include a working 
paper describing some of this work in more detail and I think that will be available 
to people who are interested in it. To answer your question, how is the gaze 
incorporated with the other sources of information? Each paragraph in the tutorial 
materials is marked as basically using xml, so the tutorial interface indicates what 
information is currently visible. The gaze tracking information is used to say, 
“Okay, the student is currently reading about topic X or now the student is 
currently looking at the screen.” We couple that with the information of what 
actions the student is actually performing. What the gaze enables us to do is to 
detect situations where the student seems to be thinking about what to do next or 
seems to be unclear about what to do next, without having to wait for them to 
make a mistake; without requiring them for the system to react. We see this as 
being potential for aborting a lot of the frustration issues that we observe when 
students aren’t sure what to do next or they do something wrong and they get the 
unexpected effect. We think that gaze will help a lot there. 

PARTICIPANT: Does the mentor change the quality of his or her 
response on the basis of that or is it just the mentor either appears or doesn’t 
appear? 
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DR. JOHNSON: Quality is primarily influenced by the internal model of 
learner self confidence and self control. Those two, as well as whether the 
student prefers to work collaboratively or not, all then can determine whether or 
not a particular intervention takes place and then also how that intervention takes 
place. If a model says that the student has a lower level of self-confidence, then 
basically the parameters of the politeness model, which promotes greater 
positive face, will be increased. So, you will see more interventions of the form, 
“Let’s try that now”, or “How about if we do this?” Things of that sort. 

DR. LEN WHITE: Part way into the session, can a learner change their 
preference in terms how attentive how they want the guidebot to be? 

DR. JOHNSON: Absolutely. And also with preferences in terms of, does 
the learner want to have this animated character at all or if the animation or 
speech be on or off. We use synthesized speech and that has a trade-off. On 
one hand, it is a way of providing feedback information to the learner without 
distracting them from their current activity, but on the other hand, you can only 
provide a limited amount of information before that becomes a tedious mode of 
delivery. We’ve noted that in some of our other studies that learners have 
different preferences in terms of what modalities that the guidebot interacting with 
them use. 
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DR. KYLLONEN: I was wondering if you could imagine a next generation 
version where you have video cameras looking for the micro-facial expressions. 
You recognize the look of contempt on the user’s face and respond to that 
appropriately. 

DR. JOHNSON: I will tell you some of the things that we’re interested in 
getting from the face and they are not things that I’ve heard talked about today. 
One, are indicators of boredom and fatigue. The vision people that I’ve been 
talking with suggest that eye closure can be an indicator of that, so that’s one 
thing we can potentially look that. And as far as the facial display issue goes, it’s 
a different situation interacting with the computer from interacting with a person. 
What we observed from looking at our videotapes is that there are some learners 
who are much more expressive in their faces than others. However, I don’t 
believe that by itself is going to be sufficient, there are definitely some learners 
that are working along and they’re just not displaying very much. And it’s not like 
they’re lying, they’re working, they’re just not emoting, but I think that’s fine. For 
the model that I described, that’s one more source of information that could be 
combined with these other sources of information. I think what we’re talking 
about is getting as many different sources of information as we can that are 
potentially uncertain and then try to combine those to get as accurate a moment- 
by-moment estimate of the learner’s state as possible. 

DR. KYLLONEN: That’s it for this session. I’ll hand this over now to Rich. 
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EMOTIONAL INTELLIGENCE: 
ASSESSMENT 


Richard D. Roberts (ETS) 


Listening. 

Learning. 

Leading. Copyright © 2004 Educational Testing Service 


DR. ROBERTS: In this session, we have assembled a panel of 
participants who will discuss various measurement approaches to the 
assessment of emotional intelligence. Our first speaker is Faba Sala, from the 
Hay Group. 
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Assessing Emotional 
Intelligence Competencies 


Fabio Sala 
Hay Group, 

McClelland Center for Research 

November 14, 2003 


ASSESSING EMOTIONAL INTELLIGENCE COMPETENCIES 

DR. SALA: Thank you very much; it’s a pleasure to be here. Of course 
thanks to Rich, for all of his efforts in helping make it happen. 

Let me begin by telling you a little bit about myself. Where I’m coming 
from: I work with the Hay Group, and we are a global human resource 
management consulting company. We help clients and businesses with a wide 
variety of business issues, problems and challenges. We essentially specialize 
and focus in on people and how to achieve results with people. I’m at the 
McClelland Center for research and innovation, named after our co-founder 
David McClelland. At the center there we are largely responsible for developing 
capability within the larger organization, so what that means is that I spend part 
of my time doing applied behavioral kinds of research, and also working directly 
with clients, applying what we’ve learned. Although we’re here talking about 
emotional intelligence, it’s a relatively small part of things that we do. 
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■ Emotional Competence Inventory (ECI) 

• Historical Context of ECI 

• Behavioral Event Interviewing 

• Emotions within Competencies 

• Psychometrics of ECI 

• Correlations with Workplace Performance 

• Knowns & Unknowns 


Where are we going? 


So let me first just explain where we are going to go today. Ultimately I’m 
going to discuss the Emotional Competence Inventory, which is the measure that 
we use to assess emotional intelligence competencies, but before that I just want 
to give you some historical context of how it was that instrument came to be 
developed. That will lead to a relatively detailed discussion of a behavioral event 
interviewing methodology, which Dave McClelland and others co-developed and 
we’ve been revising over the last 25 years. Of course we will get into some 
emotions within competencies, which came out of a lot of the work that we had 
been doing with this interview methodology. Then we’ll get into the 
psychometrics of the ECI, which has been through a couple of iterations. We’ll 
look at some correlations with work performance: we have a couple of 
correlational studies that look at the relationship between ratings on this 
assessment instrument and two really nice performance indicators in a couple of 
different settings. Then we’ll sort of wrap of with the knowns and unknowns--l’ve 
highlighted a couple but there clearly are a lot. 

One thing before moving on, in terms of assessing El competencies we 
are really going to focus on the ECI, and we’ll talk a little bit about the 
interviewing methodology. I think it’s worth noting that we have a number of ways 
of trying to get at this thing and these are just two of them... other things like 
assessment centers putting people in actual situations that will possibly arouse 
emotion like challenging situations... and code them on a relatively rigorous 
checklist and then validate our assessment using that kind of criteria, as well as 
what’s called shadowing--but that is really just walking around and watching real 
situations and following managers and executives and coding their behavior and 
trying to triangulate all these various types of assessments and figure out 
whether or not we really are capturing the essence of this person, asking whether 
what we’re capturing makes any sense and has any utility. 
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ECI in Context 


■ “Testing for Competence” 

■ Spencer & Spencer, 1993; Boyatzis, 1992 

■ Behavioral Event Interview (BEI) 

■ 3,500 Executive-Level Interviews 


So let’s get started. Ironically, the talk actually did start here; McClelland 
wrote an article in 1973 where he sort of urged folks to test for competence 
rather than intelligence. It’s nice to be back here at ETS talking about the issue. 
Spencer and Spencer and Boyatzis are a couple of authors that have worked 
closely with the McClelland Center over a number of years and sort of taken 
some of the points that McClelland argued back in 73 and started to develop this 
Behavioral Event Interview methodology. Essentially, it’s very similar to the plan- 
again critical incident assessment where you set down with someone and you 
ask them to tell stories about recent situations in the workplace--it could be 
anything. It could be about how they launched a new product, or about how they 
managed a subordinate, or anything. Then we take them through a pretty 
painstaking and rigorous process of trying to extract what actually happened. 
Ideally we like to actually go back there and be with them, and walk through this 
situation with them, observe them, stop them in the moment, as things are 
naturally unfolding and say, “what were you thinking there, why did you say what 
you just said?” 
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ECI in Context 


■ “Testing for Competence” 

■ Spencer & Spencer, 1993; Boyatzis, 1992 

■ Behavioral Event Interview (BEI) 

■ 3,500 Executive-Level Interviews 


So that’s what we try to do, that would be the ideal for us, but since we 
can’t do that we ask people to recall and reconstruct the events and with a 
relatively detailed probing process, and we like to think we do a reasonably good 
job of capturing what actually did happen. And it takes some time to develop the 
interviewing skills to get to the level of sophistication that we believe makes the 
method credible. So we’ve done this with a lot of people--at this point we have 
over 3500 interviews with executives, mostly managers, but also lower level 
people in a wide variety of organizations, a wide variety of industries, pretty 
representative in terms of male and female. During the entire process I’ve been 
revising the method. 


209 





Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Session IV: Assessment 


What are Competencies? 

■ Analytical Thinking 

■ Conceptual Thinking 

■ Interpersonal Understanding 

■ Influence 

■ Service Orientation 


This led to is a sample of what are called competencies. We have a 
sample of generic competencies and every time we do an analysis of, let’s say, a 
group of people, it’s always criterion based, so we have a sense of how effective 
that person is in their role--whether or not they’re nominated as outstanding, or 
whether or not they’re in some criterion group that indicates their effectiveness. 
So what we did after content analyzing all these interviews (which we can do 
reliably, by teaching people how to code), is glean from the narrative data what 
are called competencies--we will get to those in a minute--and identify a set of 
characteristics. 

It’s really a wide set of... it’s personalities, it’s skills, it’s traits, it’s cognitive 
abilities, and essentially to make the cut, to be a generic competency, they have 
to over a period of time be able to distinguish outstanding from average 
performance, and I mean that in a very global sense. If they show up, they have 
some values and utility for us; it’s very much empirically driven, very much 
grounded in theory. 

So there is some background. If you can imagine all this happening over a 
period of time, one of the things that seemed relatively clear is that these 
competencies seem, to a large extent, to be based in emotion, or at least part of 
what drives them is emotion-based, or what describes them, the words that are 
used to describe them, are words that you might use to describe emotion. 
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What are Competencies? 

■ Analytical Thinking 

■ Conceptual Thinking 

■ Interpersonal Understanding 

■ Influence 

■ Service Orientation 


Here are just a couple of examples of competencies that have been 
identified. Analytical and conceptual thinking are relatively straightforward and 
essentially, analytical thinking is kind of breaking things down, thinking about 
things systematically, thinking about how things are causally and linearly related, 
and then getting more sophisticated and seeing how there are multiple 
relationships between different things: A may cause B but only under condition 
D--that kind of reasoning and intellectual horse power. 

Conceptual thinking: the sense that idea that people have this ability to 
take ideas or constructs or observations that are seemingly unrelated, and then 
be able to put them together in a way conceptually that makes sense. Maybe, for 
example, a real high level of this is to create a new concept or construct nobody 
has really every thought about or identified (maybe not nobody literally but 
something like that)--sort of piecing pieces of information together that make 
sense, and that are useful and helpful for people. 

Interpersonal understanding is another competency that emerged from 
these interviews from these data, and I’ll save that, and we’ll use that one as sort 
of an illustration, what the competency looks like and feels like and essentially 
the extent to which you understand a person’s inner life, another persons inner 
life-let’s save that for a moment. 
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What are Competencies? 

■ Analytical Thinking 

■ Conceptual Thinking 

■ Interpersonal Understanding 

■ Influence 

■ Service Orientation 


Influence is the ability to get people to be persuaded by an argument--say 
I’m here and I’m trying to persuade you that, what we have is credible, and I’ll 
show you a couple of reasonable arguments, and then I’ll leave it at that. Or 
maybe I’m in a meeting with you and I’m trying to get you to do [something], to 
adopt a perspective that I’d like you to take--maybe if I’m a little more 
sophisticated I might think about how it would also suit your needs, [how it would] 
benefit the both of us and tap into that, and maybe I could get other people to 
come in and persuade. So influence is another competency that we see, that 
emerges in this context and in these situations, and there are various levels of 
sophistication. 

Service orientation is another competency, I was thinking about talking 
about that in detail, but I scrapped it. But essentially that’s not really a personality 
trait. It really is more an orientation toward giving high quality service, and you 
can see this in a wide variety in contexts. 

The competencies... really they sound like personality traits, and some of 
them are. Obviously there are also cognitive kinds of traits, certainly some 
skills... the extent to which they measure emotional intelligence as we’ll try to 
define it, I will leave that up to you. Let’s look at it in a little bit more detail. 
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related to criterion-referenced effective and/or 
superior performance in a job or situation 


Competency - I have sort of I’ve been describing them a little bit, but here 
is our formal definition: “Characteristic of an individual that causally related to a 
criterion-referenced effective and/or superior performance in a job or situation.” 
So really, all it is saying is that it is anything that potentially leads to outstanding 
performance in a job, or situation. When I say causally, I don’t meant that in the 
literal sense-that we’ve done controlled experiments and carefully tried to 
demonstrate causality-that’s not really what I mean there. But essentially it’s 
through this process of identifying of what it is that distinguishes outstanding from 
average performance, and it could be essentially anything. 

One of the interesting things that come out of this kind of analysis that we 
identify what we call uniques. And really, all that it is that we keep a very open 
mind when we are studying a sample. When we say, [for] managers in a 
particular organization or a particular industry we are not exactly sure what it is 
that leads to outstanding performance, we know, based on our experience, that 
there are certain qualities and characteristics that people tend to [have] which 
tend to make them more effective, but in a given environment we may not know 
exactly what they are, so we are open minded to these uniques. So, let’s say we 
do 20 or 30 interviews and then we have various people blindly code them to get 
transcribed and code them and so forth, and people look for uniques or 
whatever--it might be an orientation toward planning or organizing a skill; it could 
be anything. 
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And so if a pattern emerges and several different people say, hey, when I 
was reading an interview about this person, [they] seemed to be doing this a lot, 
and it seemed to be helpful--and if that is a consistent pattern that we see in that 
group, then it might be a competency that seems to distinguish outstanding from 
average performance. And if we see it frequently enough over time, over a 
number of contexts, then it makes the cut and it becomes a generic competency 
that we think adds value in a wide variety of contexts. I hope that gives you a 
sense of the concept of competencies, where they’re from, what they mean, how 
they’re used, how they were created. 
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Not to belabor, [but] let’s look at one in a little bit more in detail, back to 
interpersonal understanding. In a minute I will spend some time about the 
lessons that we’ve learned that will apply to the emotional competence inventory. 
One of the things that we’ve learned is that there are various levels of a 
competency, increasing levels of sophistication. At the most fundamental level, 
we find recognizing emotions in others: people tend to do that, and they tend to 
comment on it, so, for example, you might read body language or tone of voice, 
or you’ll notice when someone’s upset or you’ll notice when someone’s anxious 
or nervous or sad, or whatever it is, it is essentially recognizing emotions in 
others. This competency is a lot harder than it seems, so various people have 
eluded to that. Essentially what it is, is understanding the person, really 
understanding the person, and when you get to the higher levels, the long term 
reasons and issues, that’s when you really truly understand the person that you 
are interacting with, very difficult to do. So at the most basic level, we’re sort of 
reading emotions in others. 

I’m going to try to ground this in example of a story of a manager from a direct 
reports perspective. Direct report would interact with manager, and would notice 
lots of social anxiety, lots of awkwardness. When they had a conversation, the 
manager seemed to be nervous or awkward. It was relatively straightforward, but 
what the direct report also started to realize is that it made him uncomfortable as 
well. So that at the very most basic level of this competency, you might kind of 
call this competency “empathy,” maybe. 
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10 


At the next level you understand emotional verbal content. Really what 
that could be is you understand partially expressed emotions or thoughts, [or] 
that somebody might say something to you and they’re implying something but 
it’s not clear what they’re saying, and they may also be saying something else, 
and it sort of partially expressed, and it’s not clear and the extent to which you 
can tune into the inconsistencies of what they’re saying and what they’re 
expressing (non verbally for example) gives you clues about what they really 
mean. So those kinds of things would be at the next level so, you know, the 
direct report would notice the manager would get anxious whenever the direct 
report wanted to talk about performance management-essentially whenever the 
direct report would say “hey, I want to set some goals,” or “I’d like to think about 
what we can accomplish this year,” or “what the next steps in my career in terms 
of my development?” and so forth. Having those kinds of conversations 
(sometimes not directly) would cause some sort of emotion in the interaction, 
because [of] some sort of anxiety, and the person would notice it and see this 
kind of pattern. So that is a connection that the person would make and that’s at 
the second level. 
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At the next level you might understand the meaning behind that 
interaction. So: what does that interaction mean to the person? At this point the 
direct report [is] focusing on their perspective, but you really start to switch, and 
not so much focus on what you’re feeling, or what’s going on with you what it 
might mean to you, but really start to see it from the other persons perspective, 
and this is really where it gets hard. To truly understand where another person is 
coming from, it takes a lot of effort and energy, and you sort of have to suspend 
your own needs and, of course, there is all sorts of self evaluation going on--to 
what extent am I influencing this dynamic, to what extent am I perceiving 
accurately--and it gets to be a huge quagmire and at that point hopefully you are 
starting to really think about what is going on with this person, independent of 
what it means to you. Maybe you’re getting triggered now because you have 
certain goals that you want to accomplish and this person is supporting you in 
another way. So maybe you might start to develop some insights based on 
interactions, and maybe you might determine... maybe the manager might have 
a general distrust, and he likes to have control of situations. Whatever the 
appraisal might be, it’s going to be a way to help you organize all this information. 
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Finally, at the last level, longstanding underlying reasons or issues--that 
really is sort of understanding this interaction [in] context, so again, it’s hard to 
say it any other way than I’ve already described, but [this is] really understanding 
where the person is coming from--maybe you might gather information from other 
situations, from other people, and you might start to think about the larger 
organization: you realize this manager has had similar difficulties and grievances 
over his career, and [you] actually might come to learn that what happened is 
that they brought in another person that took over this person’s job, and the 
person wasn’t effectively demoted, but they got a lot of their responsibilities taken 
away, which had a lot of their identity wrapped up into it. And so you start to 
piece together all this information and you get a bigger picture where this person 
is coming from and then you begin to appreciate more some of the emotional 
manifestations that you observed that might impact you as well. That is one 
example of what one of these competencies looks like. 

This is a very rough outline, if we were to code it after reading an interview 
and a person happen to speak about something about would fall into this 
category the sort of the coding scheme would be much more complicated and 
sophisticated. 
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Okay, that is really the context. Lessons learned! So what have we 
learned from this that helps us understand how to access emotional competence 
and emotional intelligence competencies by this survey method? 

One of the things that we’ve learned is that there seems to be scaling 
going on with these competencies, they seem to have an increasing levels of 
sophistication. I don’t think that’s anything new, I’m certainly not an expert in 
personality psychology, I don’t know whole lot about personality measures, and I 
imagine that this has been observed, but I don’t know the extent to which that 
this has been systematically assessed in the way that we’ve done. The 
competencies seem to be non-linear in the sense that the distance from one level 
to the next is not always the same. So making that first level observation- 
noticing emotion-doesn’t take a whole lot for the most part, although some 
people don’t, and then getting to that second level... but the jump to that third 
level, that really takes something more, so that’s what we mean by non-linear, 
non-cumulative. 

What we’ve learned from studying competencies in this way is [that] 
sometimes within the same competency it’s not more of the same thing. It’s not 
always more of the same thing-like an influence: you make a rational argument 
to someone and you are trying to persuade them about something. 
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And we see people derail all the time, because what they do is that they 
will make this rational argument and they will be in a meeting with someone, it 
won’t go anywhere, they will both leave and nothing gets implemented [and it’s] 
wasted time, and it has not been an effective interaction. The person is banging 
their head and they go back and they get frustrated because they are not able to 
accomplish the things they need to get accomplished, and what they are doing is 
at an unsophisticated level so now they might say, “well, there’s this and this but 
that doesn’t work either,” so what we find they are non cumulative sometimes it 
takes a very different behavior. It’s still influencing but it looks very different, it’s 
just not more of the same thing. 

Another lesson learned is this idea of algorithms--this seems relatively 
obvious now that I’m reading it--you know there is more than one way to skin a 
cat. Because we really are in the business, we are really mostly interested in 
predicting performance; we work backwards, that’s essentially what we do. We 
know what we want to achieve, because business has some sort of purpose, or 
some goal, whether it’s literal, like, how do we increase sales, or whether it’s 
more vague, like, how do we insure that we have good leadership in the future. 
There is a goal, then we work backwards, that’s just how we approach it, 
obviously not a very basic science perspective. 
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So what we’ve seen is that people are successful doing very, very 
different things--the outcome is the same but obviously people are different. 
Personality dispositions are different—certainly we know that people achieve the 
very similar kinds of results in very different ways, using very different 
competencies, different behaviors, and different combinations of behaviors. One 
of the things we’ve learned is that if we’re trying to predict performance, there 
really is no one formula, even within a particular organization within the culture-- 
there are a lot of variability in terms of how people achieve those, so we have to 
be mindful of that when we’re trying to help people get there. 


221 





Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Session IV: Assessment 


Emotional Competence Inventory 

■ Self, Manager, Direct Report, Peer, Customer, 
Other Ratings 

72 items, 18 competencies, 4 clusters 

■ Behavioral items in workplace context 

■ Over 14,000 participants since 1999 


So that’s the context. Let’s get into the ECI, so the emotional competence 
inventory. It’s a 360 survey--what that means is we survey people, they do their 
self assessment--for the most part that’s just for development reasons. Primarily 
again, we’re looking to help people become more effective, let’s say as a 
manager, so it’s helpful for a person to assess themselves. And then we get 
ratings from managers, direct reports, peers, customers, others, and sometimes 
we get family members, but that’s rare. We aim to get at least 9 other raters other 
than yourself, the more the better, 12, 15 the better. Essentially, the idea is 
they’re a little more reliable, and one of the things we learned, like, for example, 
from the doctor-patient literature, when you are looking at patient satisfaction, 
which is a very important variable-you just don’t want to ask one patient, you 
want to ask a number of patients, so you make sure you get varied view points. 
So 72 items, 18 competencies, not a whole lot of items, and unfortunately there 
are only 4 items per competency-we will see how that impacts reliability in a little 
bit. 
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So the items really came from this interview method that I’ve spent some 
time talking about because we know what behaviors and skills and so forth seem 
to lead to outstanding performance at work; we sort of use that information to 
help us think about the items that might be most applicable to the content to what 
we’re most interested in. So the items reflect the workplace context and we have 
14,000 participants, and we’ve had two versions of the instrument we have about 
half and half. 7500 have taken the first version and now we have about 7500 that 
have taken this second version that seems to be a little bit better than the first-- 
I’m sort of thinking we need to revise again now and I’ll explain why. 

So the items are sort of behavioral frequency items; [each] describes a 
behavior and then asks how frequently do you see this person engaging in this 
behavior. So it might be something like “listen attentively:” do they consistently 
listen attentively, whereas maybe they don’t listen attentively very frequently, so I 
think it’s a five or six point scale. 
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So let’s just take a look at this again real quickly. Competency is 
characteristic of an individual that’s causally related to criterion, reference, 
performance, superior performance in a job or situation, so that’s really where 
we’re coming from in terms of the competency. 
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Now to make a jump to emotional intelligence competency: a competency 
that requires the recognition, understanding or integration of emotional 
information about oneself or others. So it’s this basic idea of competency and 
thinking about how it requires emotional work of some kind, either understanding 
yourself, understanding others, or some combination. 


225 






Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Session IV: Assessment 



The Conceptual Model 



Self 


Others 



I wasn’t going to show this but since Paul Ekman talked about the four 
characteristics of emotional intelligence-l'm not sure that’s exactly how he 
phrased it, but it’s nicely parallel—we can use this little model to help guide our 
understanding of what’s going on here. In the upper left is a self awareness 
orientation which the idea of being more consciously aware of emotion when you 
become emotional. The social awareness piece is becoming more sensitive to 
how others are feeling; we sort of have this belief [that] we have one piece of 
research that supports this model, but we have this belief that being more 
consciously aware of your own emotions when you’re becoming emotional helps 
you become more sensitive to how others are feeling. Then the lower left is self 
management--Paul said choosing how you behave when you are emotional, that 
is essentially what we’re talking about The more that you have some ability in 
terms of self awareness you understand how you’re more consciously aware of 
your own emotions--that’s going to help you choose to behave under the 
conditions in which you are emotional. And finally social skills: using information 
about how others feel in a constructive way, that is sort of we believe driven by 
this model that we outlined here. This is the model that helps us understand what 
really is happening and how El can help us make sense of workplace behavior. 
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Competency 

Item Text 

Accu ra te Se If Asse ssme nt 

Acknowledges own strengths and weaknesses 

Emotional Self Awareness 

Knows how their feelings effect their actions 

Self Confidence 

Presents self in an assured manner 

Achievement Orientation 

Seeks ways to improve performance 

Adaptability 

Handles unexpected demands well 

Emotional Self-Control 

Gets impatient or shows frustration 

Initiative 

Initiates actions to create possibilities 

Optimism 

Stays positive despite setbacks 

Transparency 

Acts on own values even when there is a personal cost 

Empathy 

Listens attentively 

Organizational Awareness 

Understands the organization's unspoken rules 

Service Orientation 

Matches customer or client needsto services or products 

Change Catalyst 

Is reluctant to change or make changes 

Conflict Management 

Avoids conflicts 

Developing Others 

Recognizes specific strengths of others 

Influence 

Gets support from key people 

Inspirational Leadership 

Articulates a compelling vision 

Teamwork and 

Does not cooperate with others 


Here are just some items I figured why not just look at some of them 
specifically--! outlined a few in yellow, these are the ones that really get at what 
we think is the heart of what seems like is the heart of emotional intelligence, 
which seems to be consistent among various people that talk about it. So 
emotional self control, the third one down-gets impatient or shows frustrations- 
you know that happens all the time in the work place and it can lead to negative 
outcomes, it usually derails situations, sometimes it’s an effective behavior, so 
it’s not perfect-sometimes it’s useful to get impatient or show frustration if you’re 
trying to communicate a certain message. One of the things that is clear to us is 
that emotional intelligence clearly isn’t being nice, or being good or being a 
pleasant person all the time; that is not the case at all. Certainly sometimes 
people that are highly emotional intelligent can be pretty harsh, actually. So that 
is one of the things that we’ve learned. 
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Competency 

Item Text 

Accu ra te Se If Asse ssme nt 

Acknowledges own strengths and weaknesses 

Emotional Self Awareness 

Knows how their feelings effect their actions 

Self Confidence 

Presents self in an assured manner 

Achievement Orientation 

Seeks ways to improve performance 

Adaptability 

Handles unexpected demands well 

Emotional Self-Control 

Gets impatient or shows frustration 

Initiative 

Initiates actions to create possibilities 

Optimism 

Stays positive despite setbacks 

Transparency 

Acts on own values even when there is a personal cost 

Empathy 

Listens attentively 

Organizational Awareness 

Understands the organization's unspoken rules 

Service Orientation 

Matches customer or client needsto services or products 

Change Catalyst 

Is reluctant to change or make changes 

Conflict Management 

Avoids conflicts 

Developing Others 

Recognizes specific strengths of others 

Influence 

Gets support from key people 

Inspirational Leadership 

Articulates a compelling vision 

Teamwork and 

Does not cooperate with others 


I imagine you know we might want to talk about whether or not you can 
actually observe these things in other people, like do you know whether or not a 
person acknowledges their own strength and weaknesses. They have to sort of 
verbalize that, they have to have some sort of conversation around it. One of the 
things I should say, one of the two questions we ask up front or we believe that 
are necessary for this to be at all meaningful, is to you know you really have to 
be working with this person in a significant capacity--a few times a week you 
have to have what you believe is a significant interaction, and you have to report 
that you know this person well. And I know from working with various people that 
I can’t really rate them on this because I just don’t have enough interaction with 
them, so anybody that does not score high on those two items, we don’t 
incorporate. I would love to talk about some of your thoughts on whether or we 
can actually observe that, like for example our emotional self awareness. You 
know how their feelings affect their actions, how do you know that? I think I can 
know that if I have enough opportunity to infer that, but it’s not always clear. So 
those are some of the items. 
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ECI 

Cluster 

Competency 

Total Others Rating 
(N=6,595) 

Self Rating 
(N=6,354) 

Alpha Coefficient 

Alpha Coefficient 

Self- 

Awareness 

Emotional Self-Awareness 

.87 

.71 

Accurate Self-Assessment 

.82 

.52 

Self-Confidence 

.81 

.71 

Self- 

Management 

Emotional Self-Control 

.86 

.71 

Transparency 

.74 

.52 

Optimism 

.86 

.68 

Adaptability 

.81 

.56 

Achievement Orientation 

.80 

.62 

Initiative 

.70 

.51 

Social 

Awareness 

Empathy 

.89 

.68 

Organizational Awareness 

.84 

.69 

Service Orientation 

.81 

.74 

Relationship 

Management 

Developing Others 

.89 

.73 

Inspirational Leadership 

.90 

.77 

Influence 

.81 

.63 

Change Catalyst 

.83 

.71 

Conflict Management 

.54 

.45 

Teamwork & Collaboration 

.83 

.56 


Let’s look at some of the data, here are some reliabilities, this is the most 
recent version of the ECI, the 2.0. We made a bunch of revisions that seemed to 
help, and seemed to improve it a little bit, but after this latest round, I think we 
need to do a lot more and I’ll push for that. Reliabilities are okay, the total others 
are all the peer and direct report and managers, those reliabilities seem to look 
pretty good, the self ratings have some problems-particularly conflict 
management. I think I know what’s going on, I think there are a couple of items 
that are not clear, but I don’t know. You have to keep in mind there are only 4 
items, we have 72 items in the scale, we started with a 110 with the first version, 
we just have constraints, practical constraints, we can’t have a 150 item survey, 
and have that fly—it just won’t fly. So one of the things I’d like to do is cut down 
the number of competencies cause I think some of them are secondary, second 
order competencies. Factor analysis, it’s a little shaky, there is basically one big 
factor that accounts for about 43% of the variance, so there is basically some 
global assessment of the person and what that is, we could probably disagree 
on, or debate. But it does a decent job of the 18 competencies if you try to 
confirm the 18 competencies, 14 of them 13 of them load pretty nicely, so there 
is something there, but clearly a difficult way. There is only a few measures that 
I’ve worked with that actually factor analyze so, anyway, [there is] more progress 
to be made on the instrument itself in terms of the psychometric. 
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ECI & Watson-Glaser (N=90) 


ECI 

Cluster 

Competency 

Watson-Glaser 

Self- 

Awareness 

Emotional Self-Awareness 

.04 

Accurate Self-Assessment 

-.07 

Self-Confidence 

-.13 

Self- 

Management 

Self-Control 

-.13 

Trustworthiness 

-.09 

Conscientiousness 

-.15 

Adaptability 

-.08 

Achievement Orientation 

.05 

Initiative 

-.12 

Social 

Awareness 

Empathy 

-.08 

Organizational Awareness 

-.07 

Service Orientation 

-.20 

Social 

Skills 

Developing Others 

-.23* 

Leadership 

-.21* 

Influence 

-.14 

Communication 

-.13 

Change Catalyst 

-.07 

Conflict Management 

-.14 

Building Bonds 

-.24* 

Teamwork & Collaboration 

-.19 


Murensky, C. L. (2000). The relationship between emotional intelligence, personality, critical 
thinking ability, and organizational leadership performance at upper levels of management. 
Dissertation: George Mason University. 


Here is some of the sort of construct validation stuff; I will talk about two 
studies. This is with the Watson-Glaser; you know the critical thinking ability, 
analytical reasoning test. Here is how I interpret it, it’s open to interpretation, we 
wouldn’t expect a strong relationship between these two variables. So for the 
most part, we are not seeing it, it’s roughly around zero, there seems to be a sort 
of a negative pattern there, and this is a self report of the ECl’s. So people rated 
themselves and then took this analytical reasoning test, and you can see sort of 
a little bit of a negative pattern and essentially not much relationship, so again, I 
think it’s roughly consistent with what we expect and what we find, and I’ll say 
this tentatively, in the lower quadrant there, the social skills, developing others 
and leadership and building bonds, are significantly negatively related: those that 
rated themselves higher on the ECI tended to have lower analytical reasoning 
skills on those. So one of the things that we do tend to see is that people that are 
more technically oriented have really strong analytical skills—and I know I’m 
essentially repeating the findings—but they tend to not be as good when it comes 
to managing people for example, we see people derail all the time that have 
technical skill and ability and have a background in terms of individual contributor 
background, if you know what I mean, and when they get in situations where they 
have to manage other people or be effective through other people, it’s a huge 
shift, it’s not a small shift, and a lot of people derail. It could be what’s happening 
there, but I don’t know. 
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Correlations with MBTI (N=18) 


ECI 

Cluster 

Competency 

Extraversion/ 

Intraversion 

Intuiting/ 

Sensing 

Feeling/ 

Thinking 

Perceiving/ 

Judging 

Self- 

Awareness 

Emotional Self-Awareness 

-.26 

-.62** 

-.57* 

.05 

Accurate Self-Assessment 

-.44 

-.53* 

-.64** 

-.14 

Self-Confidence 

-.40 

-.16 

-.21 

-.03 

Self- 

Management 

Self-Control 

.01 

-.57* 

-.40 

.06 

Trustworthiness 

-.29 

-.38 

-.35 

.10 

Conscientiousness 

-.08 

-.01 

-.20 

.36 

Adaptability 

-.29 

-.66** 

-.53* 

.07 

Achievement Orientation 

-.16 

-.22 

-.19 

.14 

Initiative 

-.31 

-.40 

-.47* 

.01 

Social 

Awareness 

Empathy 

-.17 

-.68** 

-.65** 

.09 

Organizational Awareness 

-.14 

-.37 

-.44 

.21 

Service Orientation 

-.23 

-.40 

-.50* 

.11 

Social 

Skills 

Developing Others 

-.32 

-.46 

-.57* 

.11 

Leadership 

-.33 

-.33 

-.56* 

.04 

Influence 

-.20 

-.41 

-.48* 

.06 

Communication 

-.24 

-.52* 

-.53* 

-.01 

Change Catalyst 

-.32 

-.39 

-.50* 

-.04 

Conflict Management 

-.23 

-.45 

-.45* 

.18 

Building Bonds 

-.36 

-.51* 

-.60** 

-.06 

Teamwork & Collaboration 

-.30 

-.60** 

-.61** 

.06 


Here are some correlations with the Myers-Briggs. So we’re measuring 
personality probably to some extent, I think it does add a little value but to use 
our consulting language, but I will leave that decision ultimately up to you. So it 
seems like there is some overlap here, the top extraversion into intuitive feeling 
and perceiving--those are in a negative direction, so those that were rated higher 
on the ECI by peers managers and direct reports--four people minimum, up to 
more than 4 people rated these people--are senior level very high level 
executives from 18 paramedics. And so people that were high in the ECI tend to 
be more extroverted; it’s not significant, you see sort of a negative pattern, you 
probably in the back can’t read the numbers, but there decent effect sizes-- 
relatively small but there seems to be something there, although not significant. 
People high on the ECI tend to a little bit more oriented toward intuition in 
perceiving information. Pretty strong relationship there; a pretty persistent 
pattern. In feeling, those high on the ECI used feeling more, used sort of the 
feeling orientation when making judgments used more personal and social 
values as opposed to logical reasoning, and not a whole lot with judging and 
perceiving. So there is clearly some overlap in terms of standard personality 
measures. 
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Self-Report: ECI & Big 5 (N=90) 


ECI 

Cluster 

Neuroticism 

Extroversion 

Openness 

Agreeable 

ness 

Conscientio 

usness 

Self- 

Awareness 

-.07 

.47** 

.28** 

.00 

.30** 

Self- 

Management 

-.20 

.24* 

.20 

-.02 

.33** 

Social 

Awareness 

-.10 

.24* 

.23* 

.03 

.21 

Social 

Skills 

-.11 

.49** 

.22* 

.08 

.39** 


Murensky, C. L. (2000). The relationship between emotional intelligence, personality, critical 
thinking ability, and organizational leadership performance at upper levels of management. 
Dissertation: George Mason University. 


Another standard personality measures is the Big 5, this is the one with 90 
executives from an oil corporation, 77 males and 13 females. Not a whole lot [of 
correlation] with neuroticism... these are both self-reports, so they rated 
themselves on ECI, and then rated themselves on the Big 5. Not a whole lot with 
neuroticism. There is certainly some overlap with extroversion, so people that are 
more extroverted tend to be higher on the ECI; a little bit with openness, a little 
bit with more openness, not a whole lot with agreeableness and consciousness. 
This wasn’t the first version of the ECI, which had a consciousness subscale, 
which is no longer on the instrument, so that correlation might be reduced a little 
bit so there is clearly a lot of overlap, or a fair amount of overlap. 
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Bass Brewers Sales Agents (N=33) 


Assessment 

1 

2 

3 

4 

5 6 

7 

8 

9 10 

1. ECI Self Rating 

- 








2. ECI Manager Rating 

.59** 

-- 







3. Overall Performance 

.62** 

.51** 

- 






4. No. New Dist. Pts. 

.40* 

.28 

.85** 

-- 





5. No. New Accounts 

.57** 

.43* 

.92** 

.78** 

-- 




6. Customer Service 

.04 

.25 

.38* 

.35* 

.16 




7. Annual Perf. Rating 

.08 

.23 

.31H 

.40* 

.15 .58** 

-- 



8. Years Bass Service 

.09 

.04 

.04 

.07 

.07 .01 

.26 

-- 


9. Career Progression 

.39* 

.23 

.51** 

.32H 

.33H .01 

.10 

.20 


10. Years of Sales Exp. 

-.27 

.07 

.02 

.04 

.01 .04 

.17 

.73** 

: .13 - 


Lloyd, M. (2001). Emotional intelligence and Bass Brewers Ltd. £><-05, £<-01 

Dissertation: Nottingham Business School. 


Finally a couple studies about criterion validity: we looked at performance 
and a couple of organizations; let’s start with this first one. This was Bass 
Brewers, really nice study, 33 people. These were area development managers. 
So within the entire U.K., these were the top 33, well, there is a smaller group, 
but these were the 33 area development managers, so they had different areas 
within the U.K. and they were responsible for generating business of course, for 
maintaining the relationships there, that they’ve already established, and looking 
to develop more business in the future. So these Bass Brewers sales agents, 
and the ECI ratings, we had both self rating and by their manager, so we just 
have one rater for the other category. And the nice thing is that there are solid 
performance metrics, it was designed to be sort of predictive, so they the 
assessments on the ECI happened six months prior to the performance numbers 
coming in and they are relatively objective. There is some subjectivity to them. 
The overall performance measure is all of the performance measures which I will 
explain in a moment all lumped together, so it’s kind of the overall. The number 
of new distribution points that really is they measure the strength of the brand in 
that region, so you want to increase brand recognition within your region. 
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Call Center Agents: Correlations 
with Performance 


ECI 

Cluster 

ECI Ratings by 

Team Leader 

Call Center Department 

Client Services 
(n=44) 

Sales 

(n=47) 

Self- 

Awareness 

Emotional Self-Awareness 

.23 

.33 

Accurate Self-Assessment 

.38 

.33 

Self-Confidence 

.61* 

.47* 

Self- 

Management 

Self-Control 

.17 

.26 

Trustworthiness 

.66* 

.53* 

Conscientiousness 

.49* 

.45* 

Adaptability 

.37 

.31 

Achievement Orientation 

.64* 

.35 

Initiative 

.58* 

.42* 

Social 

Awareness 

Empathy 

.22 

.42* 

Organizational Awareness 

.49* 

.25 

Service Orientation 

.27 

.39 

Social Skills 

Developing Others 

.30 

.30 

Leadership 

.49* 

.26 

Communication 

.41 

.32 

Influence 

.53* 

.37 

Change Catalyst 

.57* 

.43* 

Conflict Management 

.45* 

.26 

Building Bonds 

.35 

.48* 

Teamwork & Collaboration 

.44 

.41 


Nel, H. (2001). An Industrial Psychological investigation into the Relationship between Emotional 
Intelligence and Performance in the Call Centre Environment. Master’s Thesis. University of 
Stellenbosch, Department of Industrial Psychology, South Africa. 


So that’s one. Number of new accounts! That’s really sales, how many 
new distribution channels can you access, customer service, they had an outside 
organization do a survey of their customers to maintain good customer 
satisfaction, so that was the results, of the satisfaction from the customers 
perspective. Annual performance rating was done by the manager, so there is a 
little bit of a multiple common method bias here. And so that is based on mutually 
agreed-upon performance appraisal at the beginning of the year, and the 
beginning of the six months you sit down with your manager and say what do we 
want to accomplish by the end of these six months, you go back and forth, you 
negotiate, you agree upon “okay I will do this, this, and this, okay go ahead, 
touch back” and then at the end to the extent in which you achieve those goals, 
essentially like a five point rating scale. So there is some wiggle room there. 
Years of Bass service, career progression is the number of band changes, the 
number of promotions, divided by the number of years you’ve served; that is, 
your number of promotions - then finally years of sales experience and years at 
Bass. A couple of interesting things strong correlation .59 and .62 under number 
1, so [there’s a] pretty decent relationship between the ratings on the ECI and 
overall performance. 
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Call Center Agents: Correlations 
with Performance 


ECI 

Cluster 

ECI Ratings by 

Team Leader 

Call Center Department 

Client Services 
(n=44) 

Sales 

(n=47) 

Self- 

Awareness 

Emotional Self-Awareness 

.23 

.33 

Accurate Self-Assessment 

.38 

.33 

Self-Confidence 

.61* 

.47* 

Self- 

Management 

Self-Control 

.17 

.26 

Trustworthiness 

.66* 

.53* 

Conscientiousness 

.49* 

.45* 

Adaptability 

.37 

.31 

Achievement Orientation 

.64* 

.35 

Initiative 

.58* 

.42* 

Social 

Awareness 

Empathy 

.22 

.42* 

Organizational Awareness 

.49* 

.25 

Service Orientation 

.27 

.39 

Social Skills 

Developing Others 

.30 

.30 

Leadership 

.49* 

.26 

Communication 

.41 

.32 

Influence 

.53* 

.37 

Change Catalyst 

.57* 

.43* 

Conflict Management 

.45* 

.26 

Building Bonds 

.35 

.48* 

Teamwork & Collaboration 

.44 

.41 


Nel, H. (2001). An Industrial Psychological investigation into the Relationship between Emotional 
Intelligence and Performance in the Call Centre Environment. Master’s Thesis. University of 
Stellenbosch, Department of Industrial Psychology, South Africa. 


And then there’s also a strong relationship between the self and other and 
that’s unique-most of the time when we do something like this there is not a 
strong correlation between your self rating and your others rating or your 
manager rating, and there is not a strong correlation between a self rating and 
your performance. Managers do a little bit better job, but sometimes not. I think 
that one of the things we know about this organization is what is sort of called, a 
very high performance culture, they take performance very seriously, they sit 
down and they revisit their goals regularly, the managers and the sales people, 
and they meet very regularly to make sure that they are on track, that correlation 
although seems pretty high, makes some sense, it’s nice to see that years of 
service and years of sales experience really don’t relate to performance which 
sort of reaffirms this performance culture type of thing. So there are some decent 
correlations to performance in this setting. This is one with call center agents in 
South Africa and had really strong performance measures computer generated: 
your call service person and the phone call comes in and you have to manage 
that relationship, you need to give them information, you might cross sale or 
something so there is all kinds of... a handful of computer generated 
performance matrix for these people, and essentially what we see is some 
decent correlations between the performance measure and the ratings, and this 
is by their manager. 


235 































Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Session IV: Assessment 




Call Center Agents: Correlations 
with Performance 


ECI 

Cluster 

ECI Ratings by 

Team Leader 

Call Center Department 

Client Services 
(n=44) 

Sales 

(n=47) 

Self- 

Awareness 

Emotional Self-Awareness 

.23 

.33 

Accurate Self-Assessment 

.38 

.33 

Self-Confidence 

.61* 

.47* 

Self- 

Management 

Self-Control 

.17 

.26 

Trustworthiness 

.66* 

.53* 

Conscientiousness 

.49* 

.45* 

Adaptability 

.37 

.31 

Achievement Orientation 

.64* 

.35 

Initiative 

.58* 

.42* 

Social 

Awareness 

Empathy 

.22 

.42* 

Organizational Awareness 

.49* 

.25 

Service Orientation 

.27 

.39 

Social Skills 

Developing Others 

.30 

.30 

Leadership 

.49* 

.26 

Communication 

.41 

.32 

Influence 

.53* 

.37 

Change Catalyst 

.57* 

.43* 

Conflict Management 

.45* 

.26 

Building Bonds 

.35 

.48* 

Teamwork & Collaboration 

.44 

.41 


Nel, H. (2001). An Industrial Psychological investigation into the Relationship between Emotional 
Intelligence and Performance in the Call Centre Environment. Master’s Thesis. University of 
Stellenbosch, Department of Industrial Psychology, South Africa. 


We don’t really hypothesis test--that’s not really what we do; if we were to 
do some hypothesis testing, we know essentially what the requirements of that 
job are. Trustworthiness, for example, is key, credibility is key, if you are on the 
phone with someone and you feel like you can trust them and so forth you are 
more likely to be successful in that role. So we don’t have the time, but it seems 
as though if we were to hypothesis test that it would be consistent with what we 
would expect, although we didn’t literally do it, in this study. 
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Knowns & Unknowns 

■ Knowns: 

• More Basic Research 

• Predicts workplace performance 

■ Unknowns: 

• What are we measuring? 

• Theoretical Foundation 

• Psychometrics & Statisticians 


HayGroup 


So I guess to conclude I guess we know a couple of things: we certainly 
would love to do more basic research, those that have expertise in this area, the 
unknowns piece, the psychometrics and statisticians we could use some help in 
running analyses and taking advantage of the data that we have and doing more 
basic research to really understand what’s happening. We know that it is 
certainly predicts workplace performance, a couple of studies and we’ve seen it 
in a couple of other places, it predicts important moderating variables like 
organizational climate, which we know a lot about. Not really sure of what we’re 
measuring, you know, whether or not we want to call it emotional intelligence that 
is sort of a convenient way that we can capture our understandings of 
competencies and how they work. There is not really a whole a really rigorous 
theoretical foundation, but it is much more empirically driven... we’d love to have 
a chance to think about that more carefully. So those are some of the things that 
we’ve learned; we certainly need to learn a little more, and one of the things that 
I’d love to do is the study with the MSCEIT and a real standard IQ test and some 
personality measures and some observational measures and ECI and of course 
performance ratings. I would love to do that and try to partial out the various 
relationships and I encourage collaboration with anybody that’s interested. Thank 
you very much. 
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DISCUSSION 

DR. ROBERTS: Questions. 

DR. KYLLONEN: Here is the question. You said that when you did the 
analysis, the first factor accounted for 43% of it... 

DR. SALA: Yes, one big factor. 

DR. KYLLONEN: And yet you have all these scales, and I’m wondering 
whether the user is the one responsible for requesting that kind of level of 
information, or whether the user is more interested in the single factor score? 

DR. SALA: You mean when we apply it? When we get feedback? 

DR. KYLLONEN: When you report back to the user, in other words the 
data say, that you are really only measuring one thing, but the user I guess is 
saying we spent all this money on your consulting company so we need about 25 
things, is that what’s going on? 

DR. SALA: Well there’s no question that it’s helpful to them to think about 
these different kinds of behaviors, so they will get specific feedback on each of 
the competencies, and they will also look at them by each rater, so they’ll see 
discrepancies between raters and who perceives what and how that differs from 
themselves, of course technically there maybe essentially one thing that we are 
measuring, but it is extremely helpful for people to get feedback that’s hopefully 
candid from a variety of people that they work closely with, help them calibrate 
their own self perceptions. We are willing to kind of go with the error for the 
potential benefit to the individual, if that’s what you’re asking. 
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DR. IRVINE: There’s nothing like criteria in particularly if they are 
isomorphic; like the number of pints a beer drunken, which relates to the sales of 
the salesmen. It really worries about, a simple question, I assume that the 
correlations between these isomorphic hard criteria and the managers ratings is 
that correct? 

DR. SALA: I’m sorry I didn’t hear you? 

DR. IRVINE: The correlations were with managers ratings, is that correct? 

DR. SALA: Manager and self, yes. 

DR. IRVINE: So that you are not experimentally independent because the 
manager knows exactly how much these people have sold. 

DR. SALA: Of course, I said there is a common ... 

DR. IRVINE: Indeed you’ve got what we would call autocorrelation among 
these variables. Do you tell your people that, do you tell your clients that? 
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DR. SALA: Do you tell them specifically about this study? We don’t talk to 
them about this study; they are not necessarily interested. But to respond to that 
I, there is clearly a bias there, one thing I would love to do is to be able to control 
for the previous performance, because if it is predicted in terms of six months I 
assess them now and wait for the performance metrics to come in six months. As 
a manager, I have a sense of how successful they’ve been in the past. I know 
how effective they are, so that colors the influence in which I rate them. So, I 
think a nice improvement would be a control for previous performance and we 
haven’t done that. We have done a handful, 3, 4, or 5 where the performance 
ratings for example which is very much subjective, very much your manager 
rates you on a 1 to 5 scale, almost like this annual performance rating that we 
have had here at Bass, and it doesn’t correlate, so it doesn’t always correlate. 

DR. ROBERTS: Daniel Goleman, what’s his role in this, because at some 
stage, he’s listed as a consultant, he’s working with this, he’s not working with 
this, or what? 
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DR. SALA: Dan Goleman has a long history with the HayGroup 
McClelland Center because he was I think it was in the 60’s, he was one of Dave 
McClelland’s students at Harvard, and so he knew David McClelland and had 
known Richard Boyzatis, who was once I think CEO or president of McBer, so 
they knew one another, and this work really came out of the work of the 
McClelland Center. But given Dan’s ability to communicate in a way that is 
accessible to a wide variety of people that’s how, I don’t know if that answers 
your question, he got involved. He doesn’t do the basic research, he doesn’t sort 
of work directly with clients. 

DR. ROBERTS: That’s based a little bit on his theory of the actual 
dimensions of competence, or I think I heard that somewhere. 

DR. SALA: Yes, so you mean his theory of competency, I would think that 
is largely determined from what he gleaned from the work that has been done 
there. But I can’t say speak for that. 

DR. ROBERTS: Anyone else? 

DR. LEN WHITE: Maybe you can clarify this for me; McClelland, from my 
read, certainly preferred the projective technique to self-report, it is my 
understanding that there is some concern about the validity and value of self- 
report that led to the use of the behavioral event interview. What have you 
learned about the value of self-reports? Do they yield the same validity as say a 
behavioral event interview? Do you have any data like that that you can share? 
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DR. SALA: No, not specifically have we looked at that, I probably could 
actually get tons of data where people were assessed with a behavioral interview 
and then given some sort of additional assessment of them in a more sort of 360. 
But it might not work because there might be different competencies that were 
assessed and it’s more sort of convenience. One of the things I think we feel 
pretty comfortable, or clear about, is that self-reports are valid. And because they 
certainly don’t predict performance as we saw here, and tend not to correlate 
with how other people are seeing you (sometimes they do), we don’t put a whole 
lot of stock in the self-report. Indeed, we primarily use it as a way to help people 
receive their feedback and calibrate their self-perceptions. Does that help? Does 
that answer your question? 

DR. ROBERTS: Thanks Fabio. 
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PERFORMANCE-BASED EMOTIONAL INTELLIGENCE 

DR. ROBERTS: I will now hand over the podium to Jack Mayer, who will 
talk about “Performance-Based Emotional Intelligence.” 

DR. MAYER: Good afternoon, I am Jack Mayer, from the University of 
New Hampshire. I am going to talk today about performance-based approaches 
to emotional intelligence. 
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My talk today consists of four parts. 
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Overview 

• Theoretical Bases 

• Model and Measurement 

• Testing Hypotheses about Emotional 
Intelligence with the MSCEIT 

• Conclusions 


So I have four parts to this talk: 

(1) Theoretical bases, 

(2) The model and measurement we use for emotional intelligence, 

(3) Hypothesis testing with the MSCEIT, and 

(4) Then I’m actually going to spend a little time on conclusions and try to 
talk a little bit about what some other people said during the last two 
days. 
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Theory: Some of the Areas and Works 
From Which We Have Drawn 


• Artificial Intelligence 

- E.g., Abelson, 1963; Dyer, 1983; 
Picard, 1997; Sloman & Croucher, 
1981 

• Clinical Psychology 

- E.g., Sifneos, 1975; Taylor, Ryan, 
& Bagby, 1985 

• Cognition and Affect 

- E.g., Alloy & Abramson 1979; 
Averill, 1983; Bower, 1981; Isen, 
Clark, Shalker & Karp; 1988; 
Schwarz, 1990; Zajonc, 1980 

• Emotion, Appraisal, and Evolution 

- E.g., Darwin, 1872/1998; Davitz, 
1969; Ekman, 1973,2003; 
Plutchik, 1984; Roseman, 1984; 
Scherer, 1993 


• Intelligence Testing 

- E.g., Detterman, 1983; Gardner, 
1983; O’Sullivan, Guilford, & 
deMille, 1965; Sternberg, 1985; 

• Neuropsychology 

- E.g., Lane, et al., 1990 TenHouten, 
Hoppe, Bogen, & Walter, 1985 

• Nonverbal Communicatio 

- E.g., Amheim, 1974; Buck, 1984; 
Nowicki & Mitchell, 1998; 
Rosenhan & Messick, 1966 

• Philosophy 

- E.g., Calhoun & Solomon, 1984 


Barrett, Izard, 2001; Roberts, 
Zeidner, & Mathews, 2003 


Unfortunately, though I would love to, I can’t spend too much time going 
through all the amazing contributions that people, both in and outside this 
audience, have made to our original work (and continue to make to our thinking 
about emotional intelligence). However, I did want to talk about some of the 
major areas. 

When we wrote our 1990 articles (one on measurement, the other theory) 
we were drawing together disparate literatures. These included research from the 
artificial intelligence literature (articles like why should robots have emotions); 
clinical psychology work in alexythemia; Richard Lane’s Levels of Emotional 
Awareness; and studies at the interface of cognition and affect (which I certainly 
was trained in a number of years, Peter as well). We also looked at the research 
on emotional appraisal and evolution (Klaus Scherer, Paul Ekman and a number 
of others); intelligence testing (Robert Sternberg, Doug Detterman, O’Sullivan 
and Guilford, Maureen O’Sullivan spoke to you earlier today); neuropsychology 
(Richard Lane, TenHouten, and Walter); non verbal communication, and those 
people who continue to inform our work as contemporary commentators, 
including Lisa Feldman Barrett, Richard Roberts, Gerald Matthews, and so forth. 
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Emotion as Information 

• Emotion as an “object” of intelligence 

• Emotions convey information about 
relationships 

• Emotions can also operate as an auxiliary 
storage and communication system parallel 
to intelligence 

Sources: Mayer & Mitchell, 1998; Spearman, 1927; Schwarz, 1990; 
Zajonc, 1980 


My introduction has already given a lot of the theoretical ideas and that’s 
okay. And I am going to go through them a bit more quickly than I might 
otherwise, partly because I have more information about the emotionally 
intelligent individual that I would like to share with you, as identified by our test, 
the MSCEIT. 

But we agree with Paul Ekman, Klaus Scherer and many others that 
emotion conveys information and our view [is that] emotion is the object of 
intelligence, but also emotion can facilitate intelligence, which I will talk about in a 
little while. The information that emotions convey is information about 
relationships. Emotions can also operate as an auxiliary storage and 
communication system, parallel to cognition--these are ideas that a number of 
emotion researchers have put forth--we don’t claim them to be unique to our 
research team. 
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Emotions are evolved signals about 
’ .. ' al) 


Sources: Darwin, 1872/1998; Ekman, 1973 


I thought Paul Ekman was going to talk about this today, but I’ll mention it 
briefly. Emotions are evolved. Darwin came up with the idea but Paul was really 
responsible for proving Darwin’s ideas and demonstrating the generality of 
Darwin’s ideas. And another way of thinking about emotions and the information 
that they convey is through appraisal theory, which Klaus Scherer talked about 
earlier. 
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Specific Emotions Communicate 
Specific Information 


If the emotion is: 

...then the individual’s 
appraisal involves: 

Happiness 

Security; pleasure in 
joining others 

Sadness 

Loss 

Fear 

Being threatened 

Anger 

Injustice 

Sources: e.g., Ortony, Clore & Collins, 1988; Plutchik, 1994; 

Roseman, 1984; Smith & Elsworth, 1988; Scherer, 1993. 


Each emotion has a set of sub-emotions, and each sub-emotion is 
associated with a particular appraisal. This is extremely elementary, Klaus 
already got into more detail than I have here, but the idea here is fairy simple. 
Take happiness as an example. If you appraise people securely, if you have 
pleasure joining in others, that is going to be associated with happiness. 
Sadness, by comparison, is associated with senses of loss; fear is associated 
with threat; while anger relates to injustice. 
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Intelligences 

• We believe g relates to the capacity 
to carry out abstract reasoning 

- Knowing similarities and differences 

- Making generalizations 

• In addition there exist specific 
intelligences such as 

- Verbal intelligence 

- Perceptual/Organizational intelligence 

- Emotional intelligence 
Sources: e.g., Carroll, 1993; O’Sullivan, Guilford, 

& deMille, 1965; Spearman, 1927; Sternberg 
& Detterman, 1986; Wechsler, 1997. 



I am going through the intelligence material because there are lots of 
different possible perspectives on intelligence, and it may help you to have some 
sense of where we’re coming from. 

I don’t know whether this to say we or I here, so I will say the later. I 
believe that g relates to the capacity to carry out abstract reasoning, it involves 
knowing the similarities and differences among concepts, the capacity to make 
generalizations about concepts, and I believe that g provides a very good 
description of a general unitary dimension that describes a variety of cognitive 
area types of reasoning. In addition, I believe that you can model intelligences so 
it makes sense to talk about certain specific areas of intelligence: verbal 
intelligence, perceptual-organizational intelligence, spatial intelligence, and yes 
even, emotional intelligence. 
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The Scope of Emotional 
Intelligence Involves: 

• The capacity to reason with and about 
emotional signals 

• The capacity of emotion to enhance thought 


So if we are going to draw an analogy--to pick up on what one of the 
earlier speakers, said--if we’re going to draw an analogy from verbal intelligence, 
or spatial intelligence or perceptual-organizational intelligence ... if we are going 
to draw an analogy from that to emotional intelligence, then emotional 
intelligence ought to be one of two things, or both. It ought to be the capacity to 
reason with and about emotional signals, about emotional information, or, 
secondly, the capacity of emotion to enhance thought. 

Now, I think actually we can probably get pretty good agreement on that 
statement then we can move into, well okay, if you are going to do that, what are 
some of the specific areas you put under this emotional intelligence idea? And 
there we may have some disagreements; I think there is probably more than one 
valid way to talk about the specific areas. Klaus suggested some ways, Paul 
Ekman suggested some ways, but we’ll come back to that in just a moment. 
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I do want to mention briefly the popularization of emotional intelligence. In 
1995 Dan Goldman of course published his best selling book Emotional 
Intelligence, and ostensibly the book was based on our 1990 theory. 

As I’ve written in Bob Sternberg’s Handbook of Human Intelligence we 
changed our definition a bit from 1990 to 1997. 

Dan also changed our definition of emotional intelligence but in the 
opposite direction. I would like to think we made ours clearer, more focused, and 
[we gave it] more heightened utility. I think that Dan changed the definition in 
another positive way--to communicate well, communicate generally by a whole 
host of different researchers. Dan’s book on emotional intelligence started out, 
although it essentially was representing our 1990 definition, as a very broad 
representation of it indeed, including the fact that emotional intelligence was 
character, which is something we never said. 

It also made claims about El’s predictive powers that my colleagues and I 
had never made. Since then this area has come up very, very quickly. 
Sometimes we say we’ve actually constructed a whole new area of intelligence 
studies in about 15 years. In a way, we’ve gone through the whole progress of a 
century’s worth of intelligence research, and we wanted to suddenly expect 
emotional intelligence research to catch up. 
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Models and Measures: Four Branch Model of 
Emotional Intelligence 

(Mayer & Salovey, 1997; Salovey & Mayer, 1990) 


/- 

Emotional ] 

V___ 

\ 

Intelligence 




r -\ 

Managing Emotions 


f \ 

Understanding Emotions 


r \ 

Facilitating Thought 

c \ 

Perceiving Emotions 


So what is the model of the El as we see it, and what’s the way that we 
measure it? Again, I think that there are different ways to carve up the world, 
there’s not just one way. But in our reading in 1990 of the emotions research, 
non-verbal communication research by Paul Ekman, Klaus Scherer and others, 
and research more broadly in the clinical literature and elsewhere, we decided 
that there were three areas that probably could be pulled together into a potential 
domain of emotional intelligence. Then, in 1997 we made it four areas. 

And these are the four areas, and I recommend this current definition of 
emotional intelligence over the 1990 one, or along with the 1990 one. We say 
that emotional intelligence involves four areas: perceiving emotions, using 
emotions so as to facilitate thought, understanding emotions, and managing 
emotions. 
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These Four Branches Can Be 
Illustrated with the MSCEIT Test... 

• To take the test, number from 1 to 6 on a page. 

• Record your answers to the following questions... 

• Note: This is only an illustration; it is too short to 
measure emotional intelligence with confidence 
(technically, it lacks high reliability). 

• Its purpose is to illustrate how the longer tests 
work. 


And I’ll talk to you a little bit about each of those four branches, and as I 
do, I’m going to illustrate them with some test items. I apologize to about a 
quarter of the audience who I can tell just by facial, visual recognition have 
already taken this test. But for the benefit of portions of the audience, who 
haven’t had that chance, take with me some items of the MSCEIT, as I talk about 
these four areas of emotional intelligence (I think the other members of the 
audience will be patient as we do that). So to take the test, number from 1 to 6 on 
a page, if you are interested... this is what probably will be the most 
sophisticated psychometric audience I could possibly speak to, but I’ll just 
mention with just 6 items you are not going to get a reliable score, our test 
actually has 141 items to yield a reliable score. 
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Branch 1: Perception and 
Expression of Emotion 


• Ability to Identify 

Emotion in Oneself 

&A 

• Ability to Identify 

Emotion in Other 

People and Objects • 

• Ability to Express > V / 

Emotion Accurately ' / 

... 

.mi 



The first idea is that emotional intelligence involves the identification or 
perception of emotion and about the only thing that I can contribute--and it’s a 
very minor contribution on top of things that have already been said about that—is 
that the development of emotional identification we have this wonderful thing 
called empathic mirroring where parents reflect the facial expressions of their 
infants, or mirror some of the emotional expressions infants make. 

They don’t mirror it all the way; here is sort of a frantic child, the mother 
herself is not frantic, but maybe she is going oh, oh, oh, calm down a little bit. 
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MSCEIT 


How much of each emotion 
is present? 

A little A lot 

1. Sadness 1 2 3 4 5 

2. Fear 1 2 3 4 5 

3. Pleasure 1 2 3 4 5 


How much of each emotion is present in this face? Sadness from a little to 
a lot, if you’re doing this along with me, answer how much sadness is in that 
face, one is a little and five is a lot. That’s item one. Item two is the same thing, 
but do it for fear. How much fear is in that face, 1-5, five for a lot of fear. Item 3, 
how much pleasure is in that face from 1, none at all, to 5, a lot. And that’s one 
way that we have of measuring emotional identification in the MSCEIT. We will 
score [this] in a little while. 


256 



Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Session IV: Assessment 


Branch 2: Emotional Facilitation 
of Thought 


• Emotions Prioritize 
Thinking 

• Emotions are 
Available as Aids to 
Judgment 

• Mood Swings Change 
Perspective 

ILLUSTRATION: MUNCH’S “THE HOWL” 



The second branch of emotional intelligence is using emotions to facilitate 
thought. Sometimes our model is called a cognitive model, maybe that’s right, 
speakers use the expression ... I’ll leave it to you, I guess, to decide. That kind of 
terminology sort of rubs me the wrong way--l don’t think of our model as a 
cognitive model, I think it as a joint emotional-cognitive model. The same way 
that having, for example, a large vocabulary evokes a sign of high verbal 
intelligence. It’s the same thing with emotions; the more you know about 
emotions, the more you can use ... your emotional knowledge to help your 
thinking. 

So for example, if you know that when you’re sad you’re more analytical, 
and maybe a little bit less motivated, that is maybe a good time to do writing, 
because writing is an analytical task. It makes you a little bit less motivated to get 
out of your chair and do something else, [so] it might help you stay and write a 
little bit longer. So emotions can facilitate thoughts if you know about them. 
Emotions also prioritize thinking, and they can be available as an aid to 
judgment. In this particular painting, The Howl, I’m told that Munch actually felt 
pretty bad when he painted this and he used this idea of emotions as a parallel 
system, drawing on that information from his emotion system as he was painting. 
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MSCEIT 


by John D, Mayer, Peter Salovey, & David R. Caruso 


Imagine you are feeling joyful. Describe 
your feeling of joy on the following scales: 

4. Not Sweet 1 2 3 4 5 Very Sweet 

5. Very Cold 1 2 3 4 5 Very Hot 


We measure this in a kind of interesting way; I’ll show this to you just 
because I think of this as a kind of interesting task. It is called the synesthesia 
task, and we ask you to compare emotion sensations to other sensations... for 
example, imagine you are feeling joyful, describe your feeling of joy, as to how 
sweet it is. And by sweet we mean taste sweet, how sweet does it taste from 1 
not sweet at all to 5 very sweet. This is item 4. And for item 5, imagine you were 
feeling joyful and here we actually ask the people to create a small sense of joy 
within themselves and them compare it to how cold or hot you feel, one would be 
very cold, 5 would be very hot. Anyone need more time? 
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Branch 3: Understanding and 
Analyzing Emotions 


• Labeling Emotions 

• Understanding 
Emotions and 
Relationships 

• Understanding 
Complex Feelings 

• Understanding 
Transitions between 
Emotions 

Sources: Ortony et al., 1988; 
Davitz, 1969; Plutchik, 1984 



We’re actually understanding and analyzing emotion. This is another place 
were there is a tremendous amount of emotions research and appraisal research 
[that] really relates to this, but there also some stuff which is slightly disembodied 
from appraisal research which relates to emotional understanding. This is one of 
my favorite diagrams by Robert Plutchik, an early evolutionary emotions theorist, 
in which he arranged emotions in sort of emotions circle analogous to a color 
circle. I just thought this was a kind of innovative thing for him to do. The vertical 
axis [...] represents intensity, so I think he was using a fair amount of emotional 
understanding to create this diagram. 
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MSCEIT 


by John D, Mayer, Peter Salovey, & David R. Caruso 


6. What emotions might combine together to 
make “contempt?” 

A. Anger and Fear 

B. Fear and Annoyance 

C. Anger and Disgust 

D. Disgust and Joy 


Using that particular diagram as an expert criterion source, we ask the 
question, what emotions might combine together-because this is another part of 
Plutchik’s theory--what emotions might combine together to make the emotion of 
contempt. I have to be careful when I say this in front of emotion researchers, 
because some acknowledge that there are blends and some don’t. If you will 
allow me that leeway I’ll take it. So here you answer a, b, c and d. What emotions 
might combine together to make contempt: a, anger and fear, b, fear and 
annoyance, c, anger and disgust or d, disgust and joy? 
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Branch 4: 
Reflective 
Regulation of 
Emotion 

Staying Open to 
Feelings 

Reflectively Engaging 
and Detaching from 
Feelings 

Managing Emotion in 
Oneself and Others 





mmm 


Branch 4 involves regulating emotion. One of the concerns I have with the 
popularization is the emphasis that is placed on emotional control. Now 
emotional control is important, as you know when you’re raising a child it’s 
important to have the child be able to control his or her emotions to some extent, 
but there is also such a thing as over-control, and it seems like the popular 
emphasis, the media emphasis was on how does an executive really control her 
emotions at work, or really control his emotions at work, and while you do have to 
do that, no question, you also have your let your control loosen up so you can 
understand what happens when you do express your emotions, or how to 
express your emotions and so forth. 

I don’t have an item for this because it’s too long-those items are a little 
bit longer, those items actually get closest to the situational judgment tasks that I 
think ETS is interested in. 
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SCORING THE MSCEIT 





How much of each emotion 
is present? 



A little 

1. Sadness 

1 2 

2. Fear 

1 2 

3. Pleasure 

1 2 



So now I have a not so hidden motive, in asking you to take these six 
items. I want to demonstrate one form that we have of identifying a criterion for 
the correct answers to these items, which becomes an issue. 

Let me ask you, those of who took this test just now, how many people for 
sadness, said there was a little sadness in this face, number 1, raise your hand? 

(I think that Doctor Ekman had a meeting, so he had to leave the room, so you 
don’t have to worry about him judging you). How about 2, raise your hand if you 
put 2, how about 3, a few courageous souls, thank you very much, about 5 or 6 
people, how about 4, okay, that’s a lot, how about 5, a little bit less than a lot. 
Okay, so here, it being 4, we call this consensus scoring we give you credit for 4 
but we also give you a little credit for 5 cause we’re nice people and so if you put 
a 4 or 5 you get credit for that item, how about fear? How many people put 1 for 
fear? A medium amount, how about 2? Also medium, 3, far fewer, so let’s cut this 
a little quick and we’ll go to 1 or 2 for item 2. How about pleasure, how many 
people said 1 for pleasure, that’s a medium amount, 2 for pleasure, a little less, 3 
for pleasure, still 3, 4 for pleasure, 5 for pleasure... right about here we are no 
longer measuring emotional intelligence, we’re measuring social courage! 
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MSCEIT 


by John D, Mayer, Peter Salovey, & David R. Caruso 


Imagine you are feeling joyful. Describe 
your feeling of joy on the following scales: 

4. Not Sweet 1 2 3 4 5 Very Sweet 

5. Very Cold 1 2 3 4 5 Very Hot 


Imagine feeling joyful, how many people said 1 for not sweet for joyful, 2, 
3, 4, medium and 5 medium, so the answer here would be 4 or 5. Item 5, how 
many people said joy was very cold 1,2, 3, okay, 4, this is sort of medium, this is 
a little bit larger than medium and 5 a little bit dropping off, so three or four for 
how hot joyful is. Klaus, you did this all across Europe, something like what we’re 
doing right now, correct? 

DR. SCHERER: Yes. 

DR MAYER: Yes, did you, am I remembering properly that [on] one of 
those large European surveys, you sort of collected some data like this? 

DR SCHERER: Yes. 
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MSCEIT 


i>y John D, Mayer, Peter Salovey, & David R. Caruso 


6. What emotions might combine together to 
make “contempt?” 

A. Anger and Fear 

B. Fear and Annoyance 

C. Anger and Disgust 

D. Disgust and Joy 


DR. MAYER: And then finally what emotions might combine together to 
make contempt, a. anger and fear, b. fear and annoyance, c. anger and disgust, 
or d. disgust and joy. What would you say? 

AUDIENCE: C. 

DR MAYER: And so, in most places people answer, “C”, I guess it’s partly 
a function of our great translators, but we know that in Japan, in Croatia, in Israel, 
In England, in Norway, people generally answer C to this item. [...] In 
Saskatchewan, Canada, very interestingly, there is a sizeable and vocal minority 
who believe that the answer is “A” in Saskatchewan, and that is the only place 
I’ve ever been where I’ve seen that cultural difference. But there are certainly 
cultural differences we will talk about that as we go along. So this is to give you 
some idea, this is one way we score the MSCEIT with consensus scoring. 

Peter Legree is going to talk about consensus scoring later today so I’m 
not going to belabor it. But I want to say this is actually our preferred way of 
scoring the test. But other people have another way that they prefer which is with 
expert scoring, and we’ve also investigated that alternative. 
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How Was the Test Standardized? 


• Standardized on 5000 
Participants Across 
over 50 English- 
speaking data sites in: 

- Australia 

- Canada 

- India 

- South Africa 

- United Kingdom 

- United States 


• Ages 17 to 79 

• Reports matched to 
United States Census 
Data on age, gender, 
ethnicity and 
education 


So the MSCEIT now has been standardized. The original standardization 
sample is about 5000 people across 50 English-speaking data sites in a number 
of different countries. We have ages 17-79, and some of the scales have been 
matched to the United States census data. 
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TESTING HYPOTHESES 

Q: Do General Consensus and Expert 
Evaluation Converge On the Right 
Answer? 

The prediction: 

•Experts and general consensus should often converge because of the 
evolutionary-meaning hypothesis. 

•If they don’t converge, at least in part, it throws “correct answers” 
into doubt 

Sources: Legree, 1995; Roberts, Zeidner & Matthews, 2001; Mayer, 
Salovey, Caruso, & Sitarenios, 2001 


My, how can I describe... my calm equitable colleague, Frank Landy, in a 
not so recent paper, a paper around 1980 suggested that we no longer talk about 
tests’ reliability and validity, but rather we just acknowledge that we’re doing 
hypothesis testing with our tests. [...] We claim that this is content valid, that is to 
say that each of the four branches, measured each with two tasks, [is content 
valid]. We had the test called the MEIS in the late nineties, where we had 12 
tasks, which was probably better for factor analysis and stuff like that, but we 
also wanted to be practical and a twelve-branch 400 hundred item performance 
scale is not very appealing to people, so this is much more expedient. And so we 
want to ask questions about both the tests and about emotional intelligence like 
does emotional intelligence exist? Does the test work and so forth? 

And in the spirit of Doctor Landy, I’m just going to talk about some of the 
hypotheses we’ve tested as we’ve gone forward with the MSCEIT. First question 
is: How does the consensus scoring that I just went through with you compare in 
some way with what results we would get if we went out to experts and asked 
emotions experts what the correct answer was to MSCEIT? And in fact we asked 
20 members of the International Society for Research in Emotions-of which I bet 
we have a number of members here!-to please take the MSCEIT and show us 
what the correct answers were. 

Roberts, Zeidner, and Matthews and others suggested if those criteria do 
not converge, we’re in trouble. So our first question is: Can you get a right 
answer? Basically I think [what] this boils down to is, is there a right answer to 
these test items? 
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A: Yes, There is Convergence Between General 
and Expert Scoring as to the Right Answer 

• The MSCEIT rfor agreement between group consensus 
and experts ranges from r = .90 upward 

• Where Experts Beg to Differ 

- Largest differences from general group in Branches 1 
and 3 (e.g., faces and understanding) 

- This is where there is greater understanding of 
emotion 

- Emotion theories are best worked out in these areas 
Source: Mayer et al. (2003), Emotion . 


And the answer is yes, there is convergence, the convergence between 
group consensus and experts ranges .90 and upwards. We didn’t want it to be 1! 
We want experts to know something that the group doesn’t know. There are 
actually structural differences between experts and [the] general consensus. 
Experts agree with one another a lot more. 

And another small difference between experts and the group is the 
experts are particularly better on branch one, where certain people who were in 
this room just a few minutes ago have us instructed us well in how to do those 
tasks, and they are also better on branch three, where other people in this room 
have instructed so well about what emotions mean. 

Experts are great on branches one and three where the most research is, 
they are a little bit less good on two and four where there is the least--at the time 
that we did this, there was less research. 
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Q: Is the Test Reliable? 


Second question: is the test reliable? One of the things we were very 
attentive to in looking at the early nonverbal perception tests coming out from a 
number of laboratories as reviewed by Buck in 1980 (or so) was that they 
suffered from issues of reliability. 
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And I won’t go into the long song and dance about how we fix that, but we 
did fix that and so when we look at the MSCEIT for example we see that the 
overall test reliability is .93 [in] this study. Then in the branch level we have 
reliabilities of .91, .79, .80 and .83. 

Now lot of people say they’ve heard [this test] is not reliable and where is 
that coming from; well, remember, there is a lower level of tasks, two tasks for 
each branch as you can tell. The tasks are less than optimal for individual 
psychodiagnosis or psychological assessments--we don’t advise you to interpret 
the tests at the task level, we advise you to interpret it at the branch area, or the 
total test level. 
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A: Yes, There 
is Support for 
El as a Unitary 
Ability 



Normed fit index (NFI) 0.988 Relative fit index (RFI) 0.978 

Incremental fit index (IFI) 0.988 Tucker-Lewis index (TLI) 0.979 

Comparative fit index (CFI) 0.988 

RMSEA lower bound 0.115 RMSEA upper bound 0.132 


Another question, is El a unitary intelligence? That is, do various El tasks 
load on a single El structure? Now here I’ve got 8 tasks to the MSCEIT-- actually 
in the late 90’s, we were using the MEIS, which had 12 tasks—I’ll just let you 
[know] that the results here are the same as we found for the MEIS in 1991. 
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All this really says is yes, you can successfully model these 8 paths as a 
unitary ability. Now fortunately, here in 2003, I would like to think we are a lot 
more sophisticated than we were in 1980 or 1990 when we were having cut¬ 
throat arguments about is intelligence one thing, or is it several things? These 
are mathematical models and you can do it both ways, so we can model it as one 
unitary validity and that works very nicely, and we get improved fit if we model it 
alternatively as four highly correlated factors. 
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The third question is: Is there support for the four branch model of El from 
confirmatory factor analysis? 
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A: Yes, the 4 
Factor Model 
Shows a Good 
Fit 



Goodness of Fit Index (GFI) 

0.989 

Adjusted GFI (AGFI) 

0.972 

Normed fit index (NFI) 

0.977 

Relative fit index (RFI) 

0.958 

Tucker-Lewis index (TLI) 

0.964 



RMSEA lower bound 

0.042 

RMSEA upper bound: 

0.062 


Source: Mayer, et al. (2003) Emotion 


And the answer is, to continue with what I was saying, yes you can do it 
both ways, if you look at these branches as four intercorrelated areas of ability, 
you get also a very nice fit, in fact you get somewhat superior fit. You can do your 
chi-square and there is a significant improvement. I’m not showing you the chi- 
square, because you know we got 5000... actually I think this is on the first 2000 
on those first 5000 subjects... when you have 2000 subjects the chi-square will 
always be significant because it’s sensitive to N. 

So it can be scored with correct and incorrect answers. It is reliable-you 
can talk about a general emotional intelligence, you can divide it into specific 
related areas. 
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Q: Does the MSCEIT Measure 
Something New Relative to Earlier 
Tests? 

•Tests that measure similar or identical concepts 
typically correlate in the r = .65 to 1.00 range. 

•We want the MSCEIT to measure something new 
- to be independent of earlier scales - to correlate 
in the r = .00 to .35 range. 


Does the MSCEIT measure something new, relative to earlier tests? Tests 
that measure similar or identical concepts typically correlate to the .65 to 1.0 
range, we want the MSCEIT to measure something new, to be independent of 
these earlier tests... is that the case? 
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A: Apparently So. the MSCEIT is Distinct 
from all the Scales Below (N’s > 100): 


r© 

Intelligence Tests 

r= .00 to .30 

k 

Big Five Personality 
Scales 

r = .00 to .35 

k 

Self-report Scales of 

EQ, optimism, empathy 

r = .00 to .35 

Sources: Ciarrochi, Chan & Caputo, 2000; Mayer, Caruso, & 
Salovey, 1999; Roberts, Zeidner, & Mathews, 2001; Salovey, 
Mayer, Caruso,& Lopez, in press. 


And the answer is yes. You know those yin yang things, I didn’t put those 
here, I had check marks, so somehow from my Window Power Point 97 to this it 
has become yin yang. I am going to be high on Gerard Saucier’s dimension of 
spirituality for the next few minutes. 

AUDIENCE: [laughter] 

DR. MAYER: With intelligence tests, correlations are about .0 to .37 or so, 
.37 specifically between our branch 3, understanding emotions, and verbal 
intelligence. Zero between branch 4 and Ravens Matrices. Kim Barchard, who is 
now an assistant professor at the University of Nevada at Reno, used the 
O’Sullivan and Guilford scale to look at its relationship to the MSCEIT, and 
shockingly—I have to qualify this, she used only pieces of the O’Sullivan and 
Guilford and only pieces of the MSCEIT. But apparently the relationship is not all 
that high, her correlation was at the .20 level, .19 level so even correcting for 
attenuation this seems to be different than social intelligence. 
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A: Apparently So. the MSCEIT is Distinct 
from all the Scales Below (N’s > 100): 


r© 

Intelligence Tests 

r= .00 to .30 

k 

Big Five Personality 
Scales 

r = .00 to .35 

k 

Self-report Scales of 

EQ, optimism, empathy 

r = .00 to .35 

Sources: Ciarrochi, Chan & Caputo, 2000; Mayer, Caruso, & 
Salovey, 1999; Roberts, Zeidner, & Mathews, 2001; Salovey, 
Mayer, Caruso,& Lopez, in press. 


We’ve got rumors that [this is] different than practical intelligence too, but 
just rumors, I think. It is much different than the Big 5 personality scales, which 
shouldn’t surprise you—right, that’s self-report, [whereas] this is an ability 
measure, all that kind of good stuff. It’s also utterly different from self-report 
scales of emotional intelligence, or EQ as it is popularly called. Right after the 
popularization, there was a whole rash of self reported emotional scales that 
came out and we found that those have correlations with the MSCEIT between 
.35 (actually a traditional emotionally oriented empathy scale) to down to .20 
down to .0 depending upon the scale. 

Actually somebody wrote a self-report emotional intelligence scale that 
has negative correlation with the MSCEIT. So it’s really independent of a large 
number of scales that have been developed in the past. And that made us both 
happy and nervous! 
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Q: Does the MSCEIT Test Predict 
Outcomes in Organizations? 


Does the MSCEIT test predict outcomes for organizations? 
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A. Sometimes... El Helps Those Who Must 
Maintain Relationships in Organizations 



Higher-EI employed undergraduates were rated 
better-performing by supervisors 

r = .22 


Higher customer satisfaction with claims adjusters 

r = .46, 
p < .05 


Preference for a team-oriented management style 

F = 2.96, 

p < .06 


Higher-quality vision statements by groups higher 
in El, after controlling for the Big Five personality 
traits 

p < .05 

Sources: Janovics & Christianson, 2000; Rice, 1999; Moss, 2001; 
Cote, Lopes, & Salovey, in preparation 


Well, I can’t do this quickly, and I’m not sure I have enough time enough to 
do anything but give you a general idea. When you are lower in the organization 
and you have to deal with things like supervisors and customers and stuff like 
that, it really seems to matter. A group of 150 or so undergraduates, who were 
employed outside the university... Janovics and Christiansen wrote to their 
supervisors and asked them for supervisee evaluations and gave the students 
the MSCEIT and there was about a .22 correlation between the undergraduates 
and their external blind performance rating. In studies of claims adjusters, teams 
of claim adjusters with higher-than-average El, this is a small sample study, you 
can take it with a grain of salt. But they had better rated customer satisfaction, 
with a correlation of .46. 

And another study of, I believe, Yale MBAs, I am not sure about this, in a 
study of Yale MBA’s who were doing sort of simulated or maybe actual 
consultancy projects, those groups who had higher scores on the MSCEIT, 
particularly branches 3 and 4, what we call strategic El, were able to write more 
motivating and higher quality vision statements than others. 
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Q: Does El Predict Relationships 
with Others? 


Does higher El predict better relationships with others? Does lower El 
predict poorer relationships with others? 
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A: Yes, the MSCEIT Predicted these 
Positive Aspects of Relationships: 


© 

12 month old infants rated as securely 
attached in an experimental setting, have 
mothers who score higher on the MSCEIT 

p < .05 

© 

Higher El scores in adults correlate with 
higher scores on attachment scales 

r = .28, 
p < .05 

© 

Higher El scores predict more “objects of 
attachment” (e.g., photos, letters) around 
the home 

r = .30’s, 

p<.001 

Sources: Marsland & Likavec, 2003; Kafetsios, 2001; Brackett & 
Mayer, 2003 


Yes, the MSCEIT predicts a number of positive criteria concerning 
relationships. Most dramatic to me, that 12 month old infants rated as securely 
attached in the experimental strange situation have mothers who score higher on 
the MSCEIT. Higher El scores in adults correlate with more objects of 
attachment, like photos and love letters from others and friends around the 
house. 
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A: Yes...Positive Aspects of 
Relationships (Continued): 


© Higher MSCEIT scores correlate with r - .15 
higher rated social support networks ^ p 

© Higher scores on Strategic MSCEIT branches p < .05 

(understanding and management) predicted: 

■higher quality interactions with friends among 
4,500 social interactions in diaries of 100 German 
college students 

■higher rated social sensitivity in sociometric data 
among Yale University students 

Sources: Fullam, in preparation; Cote, Lopes, Salovey, & Beers, in 
press. 


People scoring higher on the MSCEIT are also better liked and are better 
evaluated by friends and colleagues, as highlighted in this slide. 
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A. Yes, Low Scores on the MSCEIT 
Predicted these Negative Aspects of 
_ Relationships: _ 


r© 

More fights, drug use 

r = .21 to .40, 1 
p < .05 


More alcohol and tobacco use 

r=. 15 to 24, 
p < .05 

© 

Higher ratings of aggression by 
peers at school 

r = .20 to .46, 

p<.001 

Sources: Formica, 1999; Trinidad & Johnson, 2001; Rubin, 2000; (N = 48). 


And lower El, this is the first finding that seems to create a pattern with the 
MSCEIT so I’m very partial to it, lower El, also predicts the number of negative 
outcomes more fights and drug abuse, more alcohol and tobacco use and higher 
ratings of aggression, by peers at school. 
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Q. What Does El Look Like in 
An Individual Case? 


As one really quick illustration of how this impacts the individual, consider 
the following case study. We asked some adolescents to tell us about a 
problematic social experience that they had had, where their peers wanted them 
to do something that they felt was conflictual for them. 
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Highest Scoring Adolescent 

• Participant 6 (16 year-old female; VIQ: 
133; EIQ: 128): Once my friends wanted 
to sneak in someone’s room and paint them 
while he slept. It began as joking around 
(“wouldn’t this be funny; could you believe 
it if?). Then it slowly evolved into dares 
(“I bet you wouldn’t,” or “I dare you to.”). 


This is just one story that a high El student told. “Once my friends wanted 
to sneak in someone’s room and paint them while he slept, it began joking 
around,” and you’ll notice I think it pulls together what a lot of other people are 
talking about, she is very socially acute, she notices the beginning of the event, 
the beginning of the emotion. “It began as joking around, ‘wouldn’t this be funny,’ 
‘could you believe it if,’ then it slowly evolved into dare, ‘I bet you wouldn’t or I 
dare you to,”’ so notice she’s not waiting until... she’s watching the evolution of 
the conflict before it gets too big. 
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Highest Scoring Adolescent 
(cont.) 

I felt like it was betraying the trust I had 
with the other person, I didn’t feel right with 
sneaking up on a sleeping person with no 
way to defend himself, and I thought doing 
this would make the person have his feelings 
hurt. I know how little pranks like this 
could really hurt someone’s feelings... 


Then she says, “I felt it was betraying the trust I had with the other person, 

I didn’t feel right, sneaking up on a sleeping person with no way to defend himself 
and I thought doing this would make the person have his feelings hurt. I know 
how little pranks like this could hurt someone’s feelings.” So here is the 
identifying emotion but also understanding how an act will bring a certain emotion 
about. 


So we asked how did you handle it, a standard question. “I told my friends 
straight out, it was a degrading thing to do and they shouldn’t be so cruel.” And 
now here in this last portion of the response is that abstract reasoning. We asked 
how would your parents have reacted, and she said, first “they would have been 
proud,” but then she switches perspective and she says, “they might also have 
said that I ruined a perfectly harmless joke.” And that to me is intelligent, to me 
she is perceiving it, she is understanding what’s going to happen, and she’s able 
to shift perspectives on it and to see it from different vantage points. 
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General Conclusions 
about Emotional Intelligence 

• EI can be defined as an intelligence 

- to perceive emotion, facilitate thought, understand emotion, 
and manage emotion 

• The MSCEIT serves as a performance measure for EI. 

• Results from the MSCEIT suggest that it does measure the 
capacity to reason with emotion, and of emotion to assist 
thinking. 

• The MSCEIT measures something new relative to the existing 
and most widely used measures of personality and intelligence. 

• High EI relates to stronger attachments; low EI to deviant, and 
often, problematic, behavior such as drug use and violence, 
among other findings. 


Now to the conclusion; first of all our research owes a debt of gratitude to 
many of those in this room, the rate of learning about emotional intelligence has 
been vary fast. The definition, our ability model has evolved and I recommend 
the 97 definition for you. 

As, Alicia Grandey, Kathryn Lively, and Aaron Ben-Ze’ev have pointed 
out, emotion is important to society, I left that out, I often started with these ideas, 
but those three contributors made such a good case for it, I didn’t need to. As the 
work of Rosalind Picard and W Lewis Johnson point out, the rules of emotional 
and social hypothesis can be worked out so carefully that computers can follow 
them. Those are substantiations that exhibit that we know a great deal about how 
emotions operate. I’m not saying we know everything, obviously we don’t, but we 
know enough to really make some good inroads here. And as Klaus Scherer 
points out, appraisal is important to emotional intelligence and Paul Ekman and 
Maureen O’Sullivan also pointed out, facial recognition is critical to emotional 
intelligence. Moreover, as Gerard Saucier, William Revelle, and Fabio Sala point 
out, there are many other personality dimensions of relevance to the individual. 
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General Conclusions 
about Emotional Intelligence 

• EI can be defined as an intelligence 

- to perceive emotion, facilitate thought, understand emotion, 
and manage emotion 

• The MSCEIT serves as a performance measure for EI. 

• Results from the MSCEIT suggest that it does measure the 
capacity to reason with emotion, and of emotion to assist 
thinking. 

• The MSCEIT measures something new relative to the existing 
and most widely used measures of personality and intelligence. 

• High EI relates to stronger attachments; low EI to deviant, and 
often, problematic, behavior such as drug use and violence, 
among other findings. 


I think of emotional intelligence and intelligence as dimensions of 
personality also, there are other dimensions of personality that are important as 
well. When we say emotional intelligence is what we say, we also mean it’s not 
optimism and it’s not niceness and it’s not persistence and it’s not zeal and it’s 
not all those other things that can go into character. We’re not saying those are 
bad things, there great things to study, but this is different; a different entity. Also 
Klaus Shearer, Karl Heider, and Gerald Saucier point out we have to watch for 
cultural differences and I didn’t get into that during this presentation; but we 
acknowledge that they exist. 

As Maureen O’Sullivan points out and as our data confirms, there is no 
relationship between self-reported El and actual El, to measure actual El you 
have to use a performance measure. More generally speaking, El can be defined 
as an intelligence: the ability to perceive emotions, facilitate thoughts, understand 
emotion and manage them. The MSCEIT serves as a reasonably good 
performance measure of emotional intelligence. And it’s convenient to use, so we 
can learn about emotional intelligence and emotionally intelligent people. The 
MSCEIT measures something new relative to the existing and most widely used 
measures of personality and intelligence. And finally high El relates to stronger 
attachment, lower El relates to deviant and also problematic behavior such as 
drug use and violence among other qualities. Thank you very much. 
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DISCUSSION 

DR. ROBERTS: We have a bit of time for questions, who would like to ask 
a question? 

DR. GRANDEY: Thank you Jack, that was great, I have a couple of quick 
questions, one was about you said that there was no relationship between self 
reported El and the ability to do El, correct? That just seems odd to me, because 
wouldn’t people who have a high ability as demonstrated performance measures 
wouldn’t they also have high understanding and awareness of their ability and 
rate themselves higher, and clearly that doesn’t work on the other end, but 
wouldn’t you see a weak relationship because of? 

DR. MAYER: I’m sure some would, and I appeal to your experience as a 
professor to answer this question, and I draw on the analogy of general 
intelligence where the relationship ranges from 0 to .35 depending on what study 
you’re talking about. You have students who are really bright, and know they are, 
you have students who are really bright but are so modest or have some 
damaged self image problem or something who think they are not very bright 
even if you know they are, and you have students who are a lot lower and think 
they are intelligent, and you have students that are not so bright and they don’t 
even understand the question so they can go anywhere with it. And that really 
limits your correlation. 
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DR. GRANDEY: Okay, and I was surprised how few studies that you were 
able to review about work place outcome and obviously from my background 
that’s what I’m interested in, is that something that you and your group are 
pursuing with the MSCEIT I mean, and I think partially because it’s a very 
involving and long measure so maybe it’s difficult to administer to a working 
sample? 

DR. MAYER: I was going to say something different about it. The test is 
two years old, it was a pre-release. So now I think we are going to see people 
bringing that in to the workplace. It takes about, it’s a commitment, it takes about 
40 minutes or so to administer. 

DR. LEN WHITE: I have two questions, one do you have a website we 
could go to learn more about MSCEIT? 

DR. MAYER: I don’t myself, my colleague David Caruso has a website 
which is emotionaliq.org for academic types there is a .com there too, he is more 
of a commercial type person so he’s got two of them. 

DR. LEN WHITE: The second question, thank you for permitting two... 
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DR. MAYER: I don’t know if you’re coming from an academic or from a 
more a consulting perspective, if you’re academic and you’re interested in 
papers, just email me and I will be happy to send you what you like and I will put 
that email up in a minute, but let me listen to your question. 

DR. LEN WHITE: I have no idea which perspective I am [laughter]... the 
other piece I would like to inquire about is in the talk earlier this morning, we 
talked, mentioned about training and staying in the area perceiving emotion... 
what is your thoughts about the trainability about what’s being measured here by 
the MSCEIT? 

DR. MAYER: Well, we said that I think from the start that there is no 
question that you can teach emotional knowledge. And I think Paul’s CD is terrific 
for that, in fact I’m going to run right out and get them. So I can teach myself. We 
do hope the test is not going to be that easily trained, but we don’t know. We like 
to think by perfecting the content of the test, by not giving away the answers and 
so forth, that although people will raise their knowledge, which we want them to 
do, we want them to raise their emotion knowledge, if that helps them (for some 
people it’s not necessary, I think). We think that there’s room for an emotional 
assessment test, an emotional achievement test. 
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DR. KATHERINE WHITE: You seem to have a lot of data on your 
measure, I’m wondering if you’ve looked at any, if you’ve tested your measure 
with older adults, if not, or if you have and have that data, I’m curious to see if 
you have the data or your prediction to whether emotional intelligence as an 
ability might behave more similar to maybe a fluid or crystallized ability across 
the life span do you have any predictions about that? 

DR. MAYER: I don’t, because I think about it and I think it could go both 
ways, I tend to think that a cohort effect that younger people are more interested 
in emotions than older people are and for that reason they maybe a little bit 
better at performing in very late years, I think of that more of the cohort effect 
than necessarily an age effect, I think it’s an empirical question I hope people like 
Derrick Isaacowitz and others who are here, who do research with an older 
population, would answer that question for us sometime soon. 

MS. MACCANN: I just wanted to say I know there is an Australian 
researcher, who has used the MSCEIT with an older population, with people in 
their 60’s and 70’s and they do tend to be of higher emotional intelligence. 

DR. MAYER: And that was with the MSCEIT? 

MS. MACCANN: That was with the MSCEIT. 

DR. MAYER: Is that right! 

DR. ROBERTS: OK, that’s it for this presentation. Thanks, Jack. 
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Measuring Knowledge Based Constructs 

Pete Legree 
& Joe Psotka 


MEASURING KNOWLEDGE BASED CONSTRUCTS 

DR. ROBERTS: It gives me great pleasure to introduce Peter Legree from 
ARI, who will discuss “Measuring Knowledge Based Constructs”. 

DR. LEGREE: My talk will describe conceptualizations that focus on 
consensual models and their role in knowledge assessment and emergent 
domains; one of which I think is emotional intelligence, although I think there are 
probably many more. I should point out that I am heavily indebted to Joe Psotka 
and Douglas Detterman, with whom I worked for the last 10 years to develop 
some of these concepts. 
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Measuring Knowledge Based Constructs 

Pete Legree 
& Joe Psotka 


These concepts relate well to the growing use of situational judgment and 
scenario based scales tests to assess knowledge and to simulate situations 
requiring expertise in performance domains such as leadership and sales, as 
well as to assess emotional and social intelligence, and for some rather specific 
purposes I think cognitive aptitude. Scales developed for these various purposes 
are similar in that they present or describe a situation and require examinees to 
indicate how they might respond or interpret the situation. These responses or 
interpretations are then analyzed to quantify performance on the tests. While 
most of these applications utilized expert groups to develop scoring standards 
(cf. Hedlund et al., 2003), sometimes scoring keys have been constructed by 
analyzing data collected from large groups of respondents who were 
knowledgeable concerning the subject domain but could not be qualified as 
experts (Legree 1995; Legree, Martin & Psotka, 2000; Legree, Heffner, Psotka & 
Martin, 2003). These individuals might be described as journeymen, but certainly 
not as certified experts in any of the traditional senses. 
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Many diverse 
opinions can be 
united into a 
better 
consensual 
whole ..many 
novices can 
combine to 
outperform an 
expert 


Happy families are all 
alike; Every unhappy 
family is unhappy in its 
own way 


In these applications, the use of non-expert groups to develop scoring 
standards was termed “Consensus Based Measurement”, and the utility of this 
method is the focus of this discussion. Scoring these scales has relied on the 
assumption that the scoring keys developed from groups of non-experts closely 
approximate those that would be from experts, and this concept is implied by this 
quote from Tolstoy and the African village proverb. 

The perspective underlying the quote and the proverb is that knowledge 
may be distributed across a population of individuals and that while it’s possible 
for individual opinions to be incorrect in numerous ways, opinions can generally 
be correct in only a single way. 

If this idea is generally valid, then consensus based measurement has the 
promise to expand the spectrum of knowledge that can be addressed in 
psychological research to include areas for which neither certified experts can be 
identified nor objective knowledge located; one of which may be emotional 
intelligence because El is an example of an emerging domain that largely lacks 
certified experts and objective knowledge. Furthermore, consensus based 
measurement has been used to score the Mayer Emotional Intelligence Scale 
(MEIS; Mayer Caruso & Salovey, 1999) and the Mayer Salovey Caruso 
Emotional Intelligence Test (MSCEIT; Mayer, Salovey, Caruso & Sitarenios, 
2003), which Jack just ran through. 
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Many diverse 
opinions can be 
united into a 
better 
consensual 
whole ..many 
novices can 
combine to 
outperform an 
expert 


Happy families are all 
alike; Every unhappy 
family is unhappy in its 
own way 


However, the notion that non-experts can be used to develop the “expert” 
knowledge required to score these instruments may be unappealing to test 
developers who are not familiar with the strengths and limitations of the 
approach, and reviewers have questioned its use and assumptions (e.g., 

Roberts, Zeidner, & Matthews 2001; Zeidner, Matthews, & Roberts, 2001; 

Schaie, 2001). So it is important to describe consensus-based measurement, 
and summarize relevant theory and data and discuss limitations of the approach. 

Most knowledge tests are based on objective knowledge or on the results 
of a job or task analysis that associates knowledge with performance in some 
domain. This approach has proven its worth in many pragmatic areas of 
personnel selection assessment and development (cf. Anastasi, 1988). Implicit 
in this approach are expectations that formal and tacit knowledge underlies much 
performance and observed behavior supports inferences concerning those 
knowledge attainments. 

Construction of knowledge scales traditionally has drawn on either an 
available and formal knowledge source (such as books written by experts; or 
pedagogical materials developed over decades of instruction and analysis) or an 
available pool of experts. 
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One basic limitation with this approach is that much knowledge is intuitive 
and tacit, and might be called mere opinion, so there may be no formal 
knowledge sources, or even experts who can provide appropriate standards. 

And in some areas, such as art, music, politics, and economics, experts may 
have or seem to have markedly different views, rationales, and evidentiary 
sources than the stratified populations of interest to researchers. Consensus 
based measurement offers unique analytic powers in these situations, and, as 
implied, this approach dovetails nicely with the use of Situational Judgment 
Tests, or scenario, based scales to assess individual abilities. 

The belief that much knowledge is experientially based and linked to 
opinion is rooted in the writings of philosophers (Plato and John Stuart Mill). And 
the concept can be identified in models of memory, such as Endel Tulving’s, that 
link different forms of memory to different evidentiary sources. Tulving’s model is 
usually used to conceptualize the development of knowledge within individuals, 
but the model also suggests that different pieces of knowledge may be 
evidenced across the memory structures of individuals. Thus one aspect of 
Tulving’s model that is often not appreciated is the notion that knowledge might 
be deduced from understanding the memories and opinions of individuals. 

The assessment of knowledge corresponding to soft, emerging domains 
such as emotional and social intelligence, where the codification and 
formalization of knowledge is only beginning, cries out for the use of these new 
technologies. These soft domains are often of considerable consequence: 
knowledge and expertise related to driving safety, leadership and social 
functioning can and does substantially impact one’s quality of life. 
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Conventional Conceptions of Knowledge Development 


Performance on a conventional test at the scale and item level across 
three levels of expertise. 


Overall scale performance distributions for a multiple-choice test 



Overall Test Performance 


Response distributions for a multiple-choice item with “C” correct 



Example item: “Which saw to use to trim moldings? (a) Azebiki 
saw, (b) Coping saw, (c) Dovetail saw, (d) Keyhole saw).” 


Our conceptualizations regarding consensus based measurement evolved 
from considering the manner with which item response distributions might 
change as a function of the expertise of respondent samples. Knowledge is 
customarily viewed as growing over levels of expertise. If a sample of 
apprentices was tracked over time, and repeatedly surveyed with standard 
knowledge items as novices, journeymen and experts, the response distributions 
described in this figure might be obtained for the test. The distributions in this 
figure illustrate individual differences as well as the expected increase in 
knowledge over levels of expertise. For an individual test item, greater 
proportions of respondents would choose the correct response over increasing 
levels of expertise, as is illustrated in lower figure (Figure 1b). 

This tendency is central to the using examinee data to approximate expert 
scoring standards. On a more personal level, instructors I’ve spoken to readily 
accept the idea that a majority rule algorithm could identify the answers for most 
multiple choice items. So it seems a little surprising that the approach is not 
more readily accepted and used. Of course, writing multiple-choice questions 
generally requires that the correct answer be known, so although this idea is 
readily accepted, at least by course instructors, it’s rarely leveraged. 
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Response Distributions and Knowledge Development 


Item response distributions across three levels of expertise that would correspond to Scenario 
based items requiring a response on a continuous Likert scale, (e.g., How much grief to 
expect for a child following the death his/her pet gerbil?). 


Differences in central tendency 


Differences in variance 


Low High 

“How much grief at loss of pet gerbil?” 



Expected differences in both central tendency and variance 


“How much grief at loss of pet gerbil?” 


High 



Novice 

Journeyman 

Expert 


However, suppose examinees were surveyed with scenario-based items 
that required a Likert rating to be associated with each item. Jack just provided 
us with five examples; I’ll give another. Examinees might be requested to 
estimate the level or intensity of an emotion associated with some event, such as 
the level of grief that a child might experience following the death of a pet gerbil. 
It’s important to appreciate that an item with this format contains the correct 
answer, and can be phrased without knowing the answer. In this example a 
sample of young children might be thought to approximate a sample of novices. 
For this type of question, the item response distributions associated with 
increased levels of expertise, for example young versus older children, might 
vary in central tendency, or in variance, or in both variance and central tendency. 

A change in central tendency might occur as children learn to expect a 
peer’s emotionality to be influenced by discrete events (e.g., when children learn 
that grief may follow the loss of a family members); changes in the central 
tendency of these types of response distributions are illustrated in Figure 2a. 

A change in variance might occur as children become more accurate in 
predicting the emotionality linked to these discrete events, in the example, 
recognizing that a gerbil is just a rodent and its death would be associated with 
only a moderate level of grief. Figure 1b illustrates a change in variance of 
response distributions associated with increased accuracy (estimating the level 
of grief). 
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Response Distributions and Knowledge Development 


Item response distributions across three levels of expertise that would correspond to Scenario 
based items requiring a response on a continuous Likert scale, (e.g., How much grief to 
expect for a child following the death his/her pet gerbil?). 


Differences in central tendency 


Differences in variance 


Low High 

“How much grief at loss of pet gerbil?” 



Expected differences in both central tendency and variance 


“How much grief at loss of pet gerbil?” 


High 



Novice 

Journeyman 

Expert 


Both these trends have relevance to understanding the growth and 
refinement of knowledge. Naive individuals lack any basis for understanding 
relationships or events, and their responses, often representing poorly formed 
cognitions, may not be sensible, sometimes indicating ignorance of even basic 
relationships, and sometimes overstating their importance. But with increasing 
degrees of sophistication, individuals will become increasingly aware and 
accurate in their understanding of relationships and events. 

This conceptualization suggests that examinees might tend to agree with 
each other to a greater extent across increasing levels of expertise for both 
conventional and scenario-based test items. Given this conceptualization, we 
recognized that estimates of the central tendency of expert response distributions 
for individual, scenario-based items would be approximated by estimates of the 
central tendency of non-expert (e.g. journeymen) response distributions when the 
growth of knowledge over expertise is associated primarily with changes in 
variance (Figure 2b). This observation also applies to conventional multiple- 
choice items (Figure 1b), but it is of little practical value because writing multiple- 
choice items requires that the correct response be known. Scenario based items 
do not always require that the correct response be specified or even known. In 
the gerbil example, the test developer need only specify the emotion, (i.e., grief) 
and allow examinees to respond to the item. 
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Response Distributions and Knowledge Development 


Item response distributions across three levels of expertise that would correspond to Scenario 
based items requiring a response on a continuous Likert scale, (e.g., How much grief to 
expect for a child following the death his/her pet gerbil?). 


Differences in central tendency 


Differences in variance 


Low High 

“How much grief at loss of pet gerbil?” 



Expected differences in both central tendency and variance 


“How much grief at loss of pet gerbil?” 


High 



Novice 

Journeyman 

Expert 


Initially, at least, we expected that changes in response distributions over 
level of expertise would be associated with changes in both central tendency as 
well as variance. This model is intermediate and is represented in Figure 1c. 

This model suggests a recursive procedure could be developed to (A) 
sequentially identify groups of individuals with increasing levels of performance 
and (B) use this information to develop valid scoring keys for scenario based test 
items. These standards would then be referred to as “consensus based 
standards” and the process as “consensus based measurement”. 
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SJT Conceptualization 

• Implicitly or explicitly describe a scenario to simulate or depict an 
event. The scenarios may be oriented towards responding to 
problems, maintaining success or interpreting an event based on 
personal experience and world knowledge. 

• Provide a list of alternatives associated with each scenario. The 
lists may include action-oriented or interpretative alternatives, or 
may allow the examinee to respond in an open-ended manner to 
describe his or her opinion and knowledge. 

• Obligate examinees to evaluate either the alternatives associated 
with the scenarios (e.g., rating the appropriateness of the 
alternatives or identifying the most relevant alternative) or to 
generate new alternatives and analyses of the scenarios in the 
case of an open-ended response. 


Situational judgment tests (SJT) are relevant for studying changes in item 
response distributions over expertise, in both central tendency and variance as 
described above. We describe situational judgment tests broadly as scenario- 
based scales with characteristics you see on that slide. 

This description is intended to be more inclusive then might be discerned 
by reviewing scales that are often described as SJTs; these scales are typically 
lists of problem-action choices. How these evaluations are scored against some 
standard and how these standards are developed is the topic of concern in this 
essay. 

The scoring standards for most existent situational judgment tests are 
developed by surveying subject matter experts to identify appropriate responses 
(cf. McDaniel, Morgeson, Finnegan, Campion & Braverman, 2001). Atypical 
procedure involves collecting ratings data for alternatives and using the mean 
expert ratings to develop a scoring standard. A percent correct agreement, a 
deviation measure, such as distance score, or a correlation of an examinee 
ratings with the scoring standard is then computed to evaluate examinee 
responses in comparison to the expert-based standards. Existing SJTs are 
potentially ideal for testing the hypotheses I’ve been suggesting to the extent that 
both expert and examinee ratings might be obtained and compared for scoring 
purposes. 
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SJT example item 


123456789 10 11 

1 I I 

Extremely Neither Inappropriate Extremely 

Inappropriate Nor Appropriate Appropriate 

You are a Work Center NCOIC. Over the past several months you have noticed that one 
of the other Work Center NCOICs in your Flight hasn’t been conducting his Common 
Task Training (CTT) correctly. Although it hasn’t seemed to affect the Flight yet, it 
looks like the Flight’s marks for CTT will go down if he continues to conduct the CTT 
incorrectly. How appropriate are the following actions? 

2 a. Do nothing since performance hasn’t yet been affected. 

7 b. Have the Work Center NCOIC meeting and tell the Work Center NCOIC who 

has been conducting training improperly that you have noticed some problems 
with the way he is training his troops. 

8 c. Tell your Flight sergeant about the problem. 

10 d. Privately pull the Work Center NCOIC aside, inform him of the problem, and 
offer to work with him if he doesn’t know the proper CTT procedure. 


Here is an SJT item developed to evaluate supervisory knowledge within 
the Army. This test item was usually scored as using a variation of a percent 
correct procedure in which the examinee identifies the most and least 
appropriate action. But the item could also be scored as a distance by 
comparing the responses of examinees and a scoring standard as shown. 
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Consensus Based Measurement Expectations 


1. High correlations between scoring standards 
based on expert and examinee data 

2. High correlations between scores based on 
these standards 

3. Adequate criterion relevance 


So given our conceptualizations regarding the changes in response 
distributions over levels of expertise, and these characteristics of scenario-based 
scales, these following relationships might be expected for scales scored with a 
census based approach. Assessing the correspondence between expert and 
examinee scoring standards requires that similar data be collected and that the 
expert sample be well described. We know of seven databases that have 
implications for these expectations, although there are probably many more. It’s 
important to recognize that most of these data do not have direct relevance to 
emotional intelligence. 

The value of this information is that the diversity of applications addresses 
the general utility of the procedure. I think the justification for these expectations 
is so strong that their assessment is better described as a parameter estimation 
exercise. So let’s review these data for implications for these conceptualizations. 
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Supervisor and Social Intelligence SJT Data 

• Three SJTs were administered to 200 AF recruits to assess supervisor 
knowledge, social interactions and social insight. 

• For the Supervisory Scale, Score Standard convergence: r=.74 (.95) 

• Score Convergence Using in AF and Army standards: r=.88 

• Factor structure supported a g-loaded model of social intelligence 


R’s 

Quartile 2 

Quartile 3 

Quartile 4 

AF Overall 

Army 

Mean Item 

SD 

Quartile 1 

.96 

.92 

.71 

.98 

.76 

2.1706 

Quartile 2 


.93 

.76 

.98 

.73 

2.3021 

Quartile 3 



.76 

.98 

.73 

2.4197 

Quartile 4 




.80 

.49 

2.5281 

AF Overall 





.74 

2.3759 


We first evaluated this conceptualization of knowledge development by 
comparing expert based scoring standards that reflected the opinions of a small 
number of subject matter experts (i.e., mean expert ratings) and the mean ratings 
for the items as computed across examinees. That Supervisor SJT described 49 
scenarios and listed a total of 202 alternatives. Each scenario described an 
interpersonal problem and presented alternatives as possible solutions to the 
problem. The test had been developed using Senior NCO’s as experts who 
rated each of the alternatives and was intended for administration to first term 
Army soldiers using a Most-Least format. However, the scale was also 
administered to a sample of AF recruits using the ratings format. Thus it was 
possible to describe the effectiveness ratings distributions for sample of NCOs 
and fairly novice recruits. 

The correlation between the expert-based scoring standard and the 
examinee mean item ratings was .74, (N=198, pc.001), and a very high 
parameter, above .95, was estimated by correcting the observed correlation for 
attenuation of the reliability of each set of observations (i.e., the mean expert and 
examinee ratings). The scores defined by the two standards correlated .88. In 
scoring these scales response distributions were equated by standardizing within 
subject. So these initial data, at least, support the conceptualization of expertise 
being associated with decreased variance. 
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Supervisor and Social Intelligence SJT Data 

• Three SJTs were administered to 200 AF recruits to assess supervisor 
knowledge, social interactions and social insight. 

• For the Supervisory Scale, Score Standard convergence: r=.74 (.95) 

• Score Convergence Using in AF and Army standards: r=.88 

• Factor structure supported a g-loaded model of social intelligence 


R’s 

Quartile 2 

Quartile 3 

Quartile 4 

AF Overall 

Army 

Mean Item 

SD 

Quartile 1 

.96 

.92 

.71 

.98 

.76 

2.1706 

Quartile 2 


.93 

.76 

.98 

.73 

2.3021 

Quartile 3 



.76 

.98 

.73 

2.4197 

Quartile 4 




.80 

.49 

2.5281 

AF Overall 





.74 

2.3759 


We tried to revise and improve the examinee-scoring standard by using a 
program to progressively select more knowledgeable examinees to produce a 
“smart examinee” scoring key, However only a marginal gain was realized by 
refining examine-based standards by using only higher scoring examinees to 
define the standard. This is because standards defined using selected samples 
of examinees were highly correlated with the inclusive examinee standard. For 
example standards based on the top quarter and the total sample correlated .98, 
which admittedly is a part-whole correlation. But it may be more surprising that 
when the data were divided by quartile, even the standard defined by lowest 
performing examinees correlated .72 with the top quartile and .49 with the NCO 
standard. Standards based on the top 3 quartiles correlated between .92 and .96 
and correlated between .72 and .76 with the NCO values. These correlations 
confirmed that the mean ratings of examinees might provide an alternate-scoring 
standard for the SJT, and this realization raised issues concerning the 
appropriateness of the two standards. From a reliability perspective, the 
examinee-based standard may be viewed as preferable because the values 
based on the opinions of a large sample of examinees (193) can be expected to 
be reliable, and because the ten years had elapsed between the collection of the 
NCO and AF data might allow shifts in social standards. 
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Supervisor and Social Intelligence SJT Data 

• Three SJTs were administered to 200 AF recruits to assess supervisor 
knowledge, social interactions and social insight. 

• For the Supervisory Scale, Score Standard convergence: r=.74 (.95) 

• Score Convergence Using in AF and Army standards: r=.88 

• Factor structure supported a g-loaded model of social intelligence 


R’s 

Quartile 2 

Quartile 3 

Quartile 4 

AF Overall 

Army 

Mean Item 

SD 

Quartile 1 

.96 

.92 

.71 

.98 

.76 

2.1706 

Quartile 2 


.93 

.76 

.98 

.73 

2.3021 

Quartile 3 



.76 

.98 

.73 

2.4197 

Quartile 4 




.80 

.49 

2.5281 

AF Overall 





.74 

2.3759 


A confirmatory factor analysis of scores for this SJT, two additional social 
intelligence scales, which could only be scored with a consensus based 
procedure, and a set of conventional cognitive aptitude tests, demonstrated the 
existence of a separate factor corresponding to a model of social intelligence. 
However the possibility that the results primarily reflected a method effect could 
not be discounted with this database. But at least for this database, and as far 
as the expectations for the scoring standards were concerned, most of the 
consensus based scoring expectations held. 
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Unobtrusive Knowledge Tests & 

Tacit Driving Knowledge Scales 

The UKT scales were designed to measure psychometric g, while 
the Driving Knowledge scales were designed to measure safe 
driving knowledge. The Driving Knowledge scales were expected 
to have low g-loadings because meta-analyses indicated a .10 
correlation between g and crash involvement. For these scales: 

• Expert standards were not available and all scales were scored 
consensually 

• Factor scores extracted from the UKT and the ASVAB correlated 
.54 (corrected for range restriction .80) 

• Factor scores extracted from the Driving Knowledge tests 
correlated with crash involvement criteria (.19) while having very 
low loadings on g, .11 (corrected for range restriction .16). 

• Only modest improvement was associated with scoring 
refinements 


In additional work, consensus based measurement was assessed by 
validating scales based on consensual measurement against conceptually 
important criteria. Specifically, we developed and validated two sets of scales: 
six Unobtrusive Knowledge Tests (UKTs) to measure general cognitive ability, 
and two Tacit Driving Knowledge Tests to assess knowledge related to driver 
safety. 

These scales were intended to have practical utility. The UKT battery was 
developed as a tool to estimate g within anonymous Internet and paper-based 
surveys and thereby provide market segmentation information. The UKTs were 
designed to appear to be opinion questions and not knowledge test items and by 
intent it was not possible to use conventional reference materials to answer the 
items. For example, one scale required individuals to estimate the frequency of 
various words in the spoken English. Thus performance on these scales could 
only be evaluated using consensus based scoring algorithms. 

The driver safety scales were intended to provide information that might 
lessen crash involvement. This latter domain is unusual because meta-analysis 
indicates only very low relationship between g and crash involvement. Thus this 
approach was applied to assess knowledge in two domains that differed greatly 
in their loadings on general cognitive ability. 
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Unobtrusive Knowledge Tests & 

Tacit Driving Knowledge Scales 

The UKT scales were designed to measure psychometric g, while 
the Driving Knowledge scales were designed to measure safe 
driving knowledge. The Driving Knowledge scales were expected 
to have low g-loadings because meta-analyses indicated a .10 
correlation between g and crash involvement. For these scales: 

• Expert standards were not available and all scales were scored 
consensually 

• Factor scores extracted from the UKT and the ASVAB correlated 
.54 (corrected for range restriction .80) 

• Factor scores extracted from the Driving Knowledge tests 
correlated with crash involvement criteria (.19) while having very 
low loadings on g, .11 (corrected for range restriction .16). 

• Only modest improvement was associated with scoring 
refinements 


These measures required individuals to respond to items using Likert 
scales, for example, estimating the frequency of words and terms used in oral 
communication or the extent to which drivers should moderate speed when 
confronted with driving hazards. Construction of these scales leveraged 
conceptualizations of incidental learning and tacit knowledge to predict and 
understand human performance. This type of knowledge and associated 
expertise is usually acquired slowly and incrementally as a result of experience 
and reflection upon those experiences in a manner consistent with Tulving’s 
memory model and as has been discussed by Sternberg (Sternberg et al., 2000). 
For these scales, neither an objective knowledge base nor experts could be 
identified to develop scoring standards. 

The UKT battery was administered to a sample of Air Force recruits. 

Factor scores extracted from this experimental battery correlated .54 with factor 
scores extracted from a conventional test battery (i.e., psychometric g); with a .80 
estimate based on the range restriction correction (Legree, Martin & Psotka, 
2000). Five of the six experimental scales also correlated significantly with 
psychometric g. This parameter estimate, .80, is typical of correlations obtained 
among cognitive ability test batteries (cf. Carroll, 1993). A Lisrel analysis of the 
corrected correlation matrix estimated a .97 path coefficient between the two 
latent factors corresponding to the Unobtrusive Knowledge and Conventional 
Test Batteries. 
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Unobtrusive Knowledge Tests & 

Tacit Driving Knowledge Scales 

The UKT scales were designed to measure psychometric g, while 
the Driving Knowledge scales were designed to measure safe 
driving knowledge. The Driving Knowledge scales were expected 
to have low g-loadings because meta-analyses indicated a .10 
correlation between g and crash involvement. For these scales: 

• Expert standards were not available and all scales were scored 
consensually 

• Factor scores extracted from the UKT and the ASVAB correlated 
.54 (corrected for range restriction .80) 

• Factor scores extracted from the Driving Knowledge tests 
correlated with crash involvement criteria (.19) while having very 
low loadings on g, .11 (corrected for range restriction .16). 

• Only modest improvement was associated with scoring 
refinements 


The tacit driving knowledge tests were administered to Army soldiers and 
correlated significantly with crash involvement data, -.11 to -.20 (Legree, Heffner, 
Martin, Psotka, & Medsker, 2002), and as expected, had only low correlations 
with measures of cognitive ability. While modest, these values generally exceed 
coefficients obtained for stable characteristics and they carry implications for 
improving driver safety. 

To summarize, these two applications demonstrate that it is possible to 
develop and validate scales with widely varying g loadings without using subject 
matter experts or objective knowledge, instead using consensus based 
measurement. Thus the data confirm the method as well as validate the scales. 
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Additional data with expert and examinee based 
standards and scores 


Scale / Source 

N: Expert/ 
Examinee 

Scoring Key 
convergence (r) 

Score 

convergence (r) 

NC021 Supervisory SJT (Porr) 

88/1891 

.89 

.95 

MSCEIT (Mayer Caruso & 

Salovey) 

21/2112 

90 

98 

Management SJT (Motowidlo) 

25/400 

.79 

-- 

TKML (Psotka & Sternberg) 

50/355 

.96 

1.00 


The above data demonstrate the efficacy of the approach, especially in 
terms of producing predictive validity, and by implication in terms of producing 
useful scoring standards. There is little doubt that consensus based 
measurement can be used to score tests developed for these unusual soft 
knowledge domains that lack formal sources of knowledge, which either may be 
very highly g-loaded, or have very minimal g loadings. It was a pleasant surprise 
that the use of a recursive-scoring algorithm was not necessary for any of these 
applications as originally expected, and this reduced the computational 
complexity of the approach. The remaining four datasets address the generality 
of the finding of agreement between expert and consensus based scores. 

The largest of these data sets corresponds to the Non-commissioned 
Officer (NCO) SJT developed to evaluate supervisory skills for senior enlisted 
soldiers. The NCO SJT described 71 problem scenarios and listed 362 actions. 
To evaluate consensus based scoring, response protocols were scored using 
both expert (N=88) and consensus (N=1891) based standards (Heffner & Porr 
2000; Porr, personal communication, July 2003). Overall performance scores 
correlated .95 and scoring standards correlated .89. 
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Additional data with expert and examinee based 
standards and scores 


Scale / Source 

N: Expert/ 
Examinee 

Scoring Key 
convergence (r) 

Score 

convergence (r) 

NC021 Supervisory SJT (Porr) 

88/1891 

.89 

.95 

MSCEIT (Mayer Caruso & 

Salovey) 

21/2112 

90 

98 

Management SJT (Motowidlo) 

25/400 

.79 

-- 

TKML (Psotka & Sternberg) 

50/355 

.96 

1.00 


Correspondence between expert and consensus based scores has also 
been demonstrated for the MSCEIT (Mayer at al., 2003), which is arguably the 
best-developed performance emotional intelligence batteries. The expert group 
corresponded to 21 members of the International Society for Research on 
Emotions, and the consensus scores corresponded to 2112 examinees who 
completed the scale. The correlation between the scores based on the two sets 
of standards was .98 and the score standards correlated .91. 
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Additional data with expert and examinee based 
standards and scores 


Scale / Source 

N: Expert/ 
Examinee 

Scoring Key 
convergence (r) 

Score 

convergence (r) 

NC021 Supervisory SJT (Porr) 

88/1891 

.89 

.95 

MSCEIT (Mayer Caruso & 

Salovey) 

21/2112 

90 

98 

Management SJT (Motowidlo) 

25/400 

.79 

-- 

TKML (Psotka & Sternberg) 

50/355 

.96 

1.00 


The last database corresponds to the Tacit Knowledge for Military 
Leadership (TKML) scale (Hedlund et al., 2003; Psotka et al., In Press). The 
TKML was designed to measure the practical, action-oriented knowledge that 
Army leaders typically acquire from experience. The TKML was developed with 
the idea that an ordered hierarchy of expertise in Military Leadership can be 
created by using the scores of Colonels as a standard and comparing them with 
U.S. Military Academy (West Point) Cadets, and U.S. Army Lieutenants, 
Captains and Majors. The scale was administered to groups of soldiers 
including: 355 Cadets, 125 Lieutenants, 117 Captains, 98 Majors and 50 
Colonels. The Colonels comprised the expert group, and this group contains the 
highest ranking soldiers and those who have served longest in the military (with 
an average of 18 years service). Comparisons of the consensus based cadet 
(355 cadets) and expert (50 colonels) scoring standards and scores provide very 
consistent results with the earlier data. The two sets of score standards 
correlated .96, and the cadet scores computed using those two standards 
correlated 1.00. Similar results were found by analyzing the data for the 
intermediate (lieutenant, captain and major) groups. Despite these correlations, 
rank was monotonically related to performance on the TKML such that higher 
ranking soldiers performed on average better than lower ranking soldiers at all 
levels. 
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TKML Ratings: SDs by Means 




33 Cadets' Mns vs SDs on Bn TKML 
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Example stem: “How much grief at loss of pet gerbil?” 
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Differences between scoring standards computed using examinee 
samples can be demonstrated using the TKML dataset, but only by isolating a 
group associated with a very low level of expertise and comparing their means 
with values based on the other groups. These relationships are comparable to 
those summarized for the Project A SJT database. For the top 25% of the 
Cadets, the correlation with the experts is 0.95. But for the bottom 25%, the 
correlation with the experts is 0.85. So by artificially restricting the examinee 
sample to the lowest quartile of the cadet sample, substantial changes in the 
standards start to emerge, but even then the correlation is still .85. 

An interesting aspect that’s associated with TKML data is that although 
the two sets of means are highly correlated, there are differences between the 
expert and examinee data such that experts agree to a greater extent when 
identify very poor responses. This is apparent by inspecting the scatter plots for 
these two distributions. 

These findings suggested that modification to the consensus based 
measurement model might be warranted such that the principle difference 
between journeymen and experts is represented in terms of increasing accuracy, 
or from the perspective of item response distributions, decreased variance 
around the item means. The transition from novice to journeyman may still be 
associated with shifts in response distributions and means because novices have 
no or little basis for their responses and their responses would be random or 
nearly random. 
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TKML Ratings: SDs by Means 
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We expect the scatter plot differences reflect censoring in the experiences 
available to the experts in comparison to the students. 

The cadets do relatively well on picking out the good alternatives, but 
appear to have real difficulty discerning how bad the irrelevant actions are. 

They think the bad items are better than the experts do, and their standard 
deviations (SDs) for the bad items are much higher than the experts’. 

As in Tolstoy’s magnificent phrase, there are so many different ways to go 
wrong that they appear to be overwhelmed by the enormity of all the possibilities. 

Also, our personal experiences and formal knowledge learned from others 
are full of advice and experience about the right ways of doing things, but 
impoverished in the experience of what not to do. 
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Conceptualizing Consensus Based 
Measurement: Summary 

• Expert and Examinee standards will be highly correlated when 
the distributions of precipitating events are similar for these 
groups, 

• This is equivalent to the expectation that when exposure to 
experiences and declarative knowledge is similar over levels of 
expertise, r(expert,truth)>r(journeyman,truth)>r(novice,truth) 

• Disagreement is expected when the distributions of exposure to 
either declarative knowledge or experiences is censored 

- Declarative Knowledge Expectations: Alcohol and crash 
involvement, Urban crime 

- Proscribed Experiences: Teen Smoking, Sexuality & Alcohol 
expectations 


To understand consensus based scoring, it is useful to consider that for 
most knowledge domains, and especially for procedural knowledge domains, 
knowledge accumulates as the result of experience (cf. Anderson & Lebiere, 
1998). As a greater range of events is experienced, greater levels of knowledge 
and associated skills will be acquired, and reactions to a new event or situation 
may reflect increasing levels of sophistication. It follows that the same reasoning 
applies to exposure to declarative knowledge as outlined by Tulving, although 
exposure to declarative knowledge in academic settings may be much more 
systematic while exposure to declarative knowledge through mass 
communications may be much more repetitive and possibly biased. 

In the absence of systematic biases in exposure to experiences or 
declarative knowledge, components of the novices’ thinking should be in error in 
different ways, but the components that are on the road to expertise should be 
similar. 

When presented with a situation to analyze, novices will have little basis 
for their opinions, and they will frequently disagree among themselves as well as 
with experts. Disagreement among novices is expected because the knowledge 
and cognitive structures associated with an individual novice will reflect either the 
action of a few unique experiences or the actions of experiences that have 
marginal relevance to the presented situation. Thus novices will reference 
different experiences and expectations, and their opinions will tend to be 
inconsistent, both among themselves as well as with experts. 
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Conceptualizing Consensus Based 
Measurement: Summary 

• Expert and Examinee standards will be highly correlated when 
the distributions of precipitating events are similar for these 
groups, 

• This is equivalent to the expectation that when exposure to 
experiences and declarative knowledge is similar over levels of 
expertise, r(expert,truth)>r(journeyman,truth)>r(novice,truth) 

• Disagreement is expected when the distributions of exposure to 
either declarative knowledge or experiences is censored 

- Declarative Knowledge Expectations: Alcohol and crash 
involvement, Urban crime 

- Proscribed Experiences: Teen Smoking, Sexuality & Alcohol 
expectations 


In contrast, experts will generally have well-developed, mature knowledge 
structures reflecting broad, extensive sets of experiences and exposures to 
declarative knowledge. While each expert will have a slightly different set of 
experiences, these sets will largely overlap across individual experts, and with 
increasing levels of expertise, knowledge structures and related opinions will 
become progressively more consistent. Journeymen with partially developed 
and varying levels of expertise will agree at a moderate level both among 
themselves and with experts, and this moderate level of agreement is based on 
developing cognitive structures that reflects a modest but not extensive array of 
experience. 

Because procedural knowledge is experientially based and because these 
experiences are generally dependent on the occurrence of real-world events, 
various journeymen may have different types of experiences and knowledge, 
although much of this knowledge will be most relevant to those situations that 
frequently occur. It follows that the breadth of experience associated with a 
single expert, while more extensive than that of an individual journeyman, will 
often be exceeded by the variety of experiences associated with a substantial 
number of journeymen. The implication of this view for consensus based 
measurement, as well as for other knowledge engineering applications, is that 
more information might be present in the knowledge structures of a large number 
of journeymen than a small number of experts. 


316 




Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Session IV: Assessment 


Conceptualizing Consensus Based 
Measurement: Summary 

• Expert and Examinee standards will be highly correlated when 
the distributions of precipitating events are similar for these 
groups, 

• This is equivalent to the expectation that when exposure to 
experiences and declarative knowledge is similar over levels of 
expertise, r(expert,truth)>r(journeyman,truth)>r(novice,truth) 

• Disagreement is expected when the distributions of exposure to 
either declarative knowledge or experiences is censored 

- Declarative Knowledge Expectations: Alcohol and crash 
involvement, Urban crime 

- Proscribed Experiences: Teen Smoking, Sexuality & Alcohol 
expectations 


From a mathematical perspective, the correlation of knowledge between 
individual A and individual B can be conceptualized as the product of the 
correlation of individual A with the 'truth' and of individual B with the 'truth'. As 
individuals A and B become more knowledgeable and their opinions more 
“truthful”, their opinions and responses will become more highly correlated (cf. 
Romney & Weller, 1984). 

For many domains progression is reasonable, but for some domains 
exposure to experiences or knowledge may be heavily censored. This might 
occur when certain activities are proscribed from segments of the population, for 
example when youth develop unrealistic expectations concerning alcohol or 
social interactions, when knowledge that is repeatedly conveyed through mass- 
communications or within academic settings is one-sided or biased. 

These learning theories are most relevant to understanding consensus 
based measurement when cognitive structures and related knowledge reflect the 
experience of largely unpredictable events, as does much procedural and tacit 
knowledge. In contrast, academic knowledge reflects more formal instruction, 
which is often structured to provide a systematic, highly ordered set of 
experiences based on objective information, and the surveying of students on 
topics not yet covered is unlikely to identify much information. Flowever most of 
these domains correspond to procedural knowledge, be it incidental or tacit, as 
do many SJTs (cf. McDaniel et al., 2001), and we suspect, many soft, poorly 
defined domains. 
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Conceptualizing Consensus Based 
Measurement: Summary 

• Expert and Examinee standards will be highly correlated when 
the distributions of precipitating events are similar for these 
groups, 

• This is equivalent to the expectation that when exposure to 
experiences and declarative knowledge is similar over levels of 
expertise, r(expert,truth)>r(journeyman,truth)>r(novice,truth) 

• Disagreement is expected when the distributions of exposure to 
either declarative knowledge or experiences is censored 

- Declarative Knowledge Expectations: Alcohol and crash 
involvement, Urban crime 

- Proscribed Experiences: Teen Smoking, Sexuality & Alcohol 
expectations 


Thus cognitive theories related to the acquisition of procedural knowledge 
support the contention that the opinions of a large number of journeymen can be 
used to approximate those opinions of a smaller number of experts for these 
types of domains, and this notion is the heart of consensus based measurement. 
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Conceptualizing Consensus Based 
Measurement: Implications 

• Supports scale development in emerging domains often lacking 
experts, such as Emotional Intelligence 

• Provides economy to test development 

• Explicitly invokes the concept of disagreement to understand 
knowledge structures 


Much knowledge represents the convergence between many perspectives 
and truth is commonly believed to exist at the intersection of these perspectives. 
The perspective that knowledge is rooted in widely diverse opinion is reflected in 
Tolstoy’s observation that "Happy families are all alike; every unhappy family is 
unhappy in its own way", and from a cross-cultural perspective, the African 
proverb, “It takes a village to raise a child.” The success of these institutions and 
the relevance of these statements reflect the notion that knowledge can be 
distributed over individuals, and is consistent with use and development of 
technologies to identify this type of knowledge and its evidentiary sources for 
emerging fields such as social and emotional intelligence. 

Consensual scoring has several important implications for studying 
individual differences. First, the approach allows the construction and scoring of 
scales for knowledge domains for which experts do not exist, or cannot be easily 
identified. This allows an expansion of the domains for which knowledge tests 
may be developed, an expansion beyond traditional formal domains into 
everyday knowledge areas that are meaningful and important in our daily lives. 
Thus consensus based scoring allows the assessment of knowledge domains 
which have not been traditionally addressed in psychological or educational 
research, and broadens the domain of psychological assessment and 
intelligence research into horizontal aspects of intelligence, one of which may be 
emotional or social intelligence. This perspective is consistent with theories of 
implicit and tacit knowledge acquisition and relates well to conceptualizations of 
social knowledge. 


319 




Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Session IV: Assessment 


Conceptualizing Consensus Based 
Measurement: Implications 

• Supports scale development in emerging domains often lacking 
experts, such as Emotional Intelligence 

• Provides economy to test development 

• Explicitly invokes the concept of disagreement to understand 
knowledge structures 


A second important implication is that consensus based measurement 
provides economy to test development. The approach allows questions to be 
posed, answered and scored without the correct responses known a priori. Thus 
the scale development cycle is shortened because expert responses are not 
required to construct scoring standards. In addition, costs associated with the 
production of scoring standards and rubrics are minimized because expert 
judgments can be expensive to collect while the examinee data are incidental to 
scale administration. And the use of Likert scales allows distances to be 
computed at the item level, thus providing ratio as opposed to dichotomous 
information for each item. 

Third, consensus based scoring has the potential to allow the same 
protocol to be scored against multiple standards. This approach could be useful 
in studying controversial domains associated with groups that may adopt 
different perspectives. This approach might relate well to understanding 
controversial views differing over gender, political affiliation, race, age or sexual 
orientation or in identifying the basis for competing theories to explain some 
phenomenon. This application has similarity to multi-dimensional scaling 
concepts, but the existence of different scoring standards would imply different 
types of knowledge structures and evidentiary sources correspondent to a 
shared topic. So the approach invokes the notion of disagreement and 
inconsistency in the coherence of knowledge structures. Ill-defined domains are 
characterized by disagreement even among experts. 
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To describe consensus based measurement and to summarize data 
describing its effectiveness and utility were two goals of this discussion. But the 
initial model was more descriptive than theoretical, and the concept that expert 
knowledge can be approximated by surveying large numbers of non-experts 
must have some limitations. To understand consensus based scoring, it is useful 
to consider that for most knowledge domains, and especially for procedural 
knowledge domains, knowledge accumulates as the result of experience (cf. 
Anderson & Lebiere, 1998). As a greater range of events is experienced, greater 
levels of knowledge and associated skills will be acquired, and reactions to a new 
event or situation may reflect increasing levels of sophistication. It follows that 
the same reasoning applies to exposure to declarative knowledge as outlined by 
Tulving, although exposure to declarative knowledge in academic settings may 
be much more systematic while exposure to declarative knowledge through mass 
communications may be much more repetitive and possibly biased. 

In the absence of systematic biases in exposure to experiences or 
declarative knowledge, components of the novices’ thinking should be in error in 
different ways, but the components that are on the road to expertise should be 
similar. 
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When presented with a situation to analyze, novices will have little basis 
for their opinions, and they will frequently disagree among themselves as well as 
with experts. Disagreement among novices is expected because the knowledge 
and cognitive structures associated with an individual novice will reflect either the 
action of a few unique experiences or the actions of experiences that have 
marginal relevance to the presented situation. Thus novices will reference 
different experiences and expectations, and their opinions will tend to be 
inconsistent, both among themselves as well as with experts. 
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In contrast, experts will generally have well-developed, mature knowledge 
structures reflecting broad, extensive sets of experiences and exposures to 
declarative knowledge. While each expert will have a slightly different set of 
experiences, these sets will largely overlap across individual experts, and with 
increasing levels of expertise, knowledge structures and related opinions will 
become progressively more consistent. Journeymen with partially developed 
and varying levels of expertise will agree at a moderate level both among 
themselves and with experts, and this moderate level of agreement is based on 
developing cognitive structures that reflects a modest but not extensive array of 
experience. 
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Because procedural knowledge is experientially based and because these 
experiences are generally dependent on the occurrence of real-world events, 
various journeymen may have different types of experiences and knowledge, 
although much of this knowledge will be most relevant to those situations that 
frequently occur. It follows that the breadth of experience associated with a 
single expert, while more extensive than that of an individual journeyman, will 
often be exceeded by the variety of experiences associated with a substantial 
number of journeymen. The implication of this view for consensus based 
measurement, as well as for other knowledge engineering applications, is that 
more information might be present in the knowledge structures of a large number 
of journeymen than a small number of experts. 


324 














Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Session IV: Assessment 


DISCUSSION 


DISCUSSION 

DR. ROBERTS: Questions. 

DR. STRICKER: I have been mediating over what seems to me as 
psychometric paradox underlined consensus scoring, namely when you do the 
consensus scoring you hope that nearly everybody or certainly a lot of people will 
agree on a particular response. 

DR. LEGREE: Not really, I think what you’re hoping on, what you’re 
hoping for is that your disagreements will be inconsistent. 

DR. STRICKER: Maybe that’s the answer cause because I was going to 
say then, when you then administer that test operationally to the same kind of 
people who you use for your consensus judgments, you’d expect that most of 
those people would give the same answer in which case there wouldn’t be much 
variation in the items and wouldn’t be much in the way of individual differences, 
but I think you answered the question. 

DR. LEGREE: I think I do, I can repeat the answer. They’re scored as 
distances from the means, so lower levels of expertise was associated with 
greater variance around each mean. 


325 




Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Session IV: Assessment 


DISCUSSION 


DR. SAUCIER: Let me ask a practical question, I have another abstract 
one that I’ll deal with if we have time. Jack Mayer was using consensus based 
scoring and in our exercise he was identifying modal responses then you got a 
basically a plus score for it if you had the modal response and you didn’t get any 
points I guess if you didn’t have the modal response and that was based on 
some kind of Likert scale? When people use Likert scales some people tend to 
use the extreme to the scale, some people tend to use the middle, some people 
like to go high, some people like to go low. So my question is, 

DR. LEGREE: You’d like to know how you’d adjust for that? 

DR. SAUCIER: Yeah, well I’m wondering if it’s actually better to score 
them by looking at the correlation to the consensus judgment rather than 
identifying how close it is to the modal response. 

DR. LEGREE: That’s exactly how the TKLM was scored. Depending 
upon the application we’ve transformed within subjects responses so that the 
responses for all the individuals have the same mean and standard deviation. In 
general when we’ve run analysis using simple distances or distances computed 
using the transform values there is a very high correlation. We tend to obtain 
slightly higher coefficients in terms of factor analysis and validity data with the 
more sophisticated approach which really piggy backs on your question, and yes 
that is correct and there is a solution. 
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DR. KYLLONEN: I have a question: it seems that your procedure will 
always work unless there is some kind of bias. 

DR. LEGREE: I’d like to think so. 

DR. KYLLONEN: Unless there is some sort of bias that experts or novices 
see the world in a different way than experts do. 

DR. LEGREE: And I think that is correct. 

DR. KYLLONEN: And so I guess the question is this. Is there a diagnostic 
procedure for you to use so you’d be able to detect whether there might be a 
systematic bias among novices? Or among the population that would be different 
from experts without collecting all the data and going through the tedious process 
of comparing the two scoring keys? 
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DR. LEGREE: Well, I presume there are two variations on your question. 
One addresses item level statistics, in our analysis we have found occasionally 
poor items, I will give you an example, in one particular scale we asked, it was 
actually the passive driving safety scales, passive driving knowledge scales, we 
asked examinees to estimate the proportion of crashes which involved drunk 
drivers and the response distribution simply wasn’t reasonable. The medium 
response was 75% and some small portion of response indicated all crashes 
involved drunk drivers. And it came as a real surprise, a very non sensible 
response, what we found out was that prior to any long weekend there is a 
mandatory safety briefing given to all soldiers and almost as a matter of routine, 
officers hit drunk driving so there is one message that is told at every safety 
briefing is don’t drink and drive, they don’t mention possible relationships 
between being tried and driving or being angry and driving. As a consequence, 
the examinees overestimated the involvement of drunk drivers and crashes and 
the item acted very poorly in the factor analysis. In terms of the other questions, 
suppose there was a domain which was entirely biased, in the sense of 
comparing the experiences of experts and novices. That is possible, and we 
haven’t worked out methods to identify those domains. 
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DR. LEN WHITE: There seems to me that, and maybe that’s what you’re 
getting at; does your data suggest that we may need to be more careful about 
who we call an expert, or who is called an expert? How does one know if you 
have an expert? 

DR. LEGREE: I think it would be wise to be more careful in terms of who’s 
identified as an expert, but for many domains I’m not sure it really matters. There 
is a natural progression from apprentice to journeyman to expert and for the most 
part, experts are simply more accurate and more sensible in their understanding 
and expectations. 

DR. LEN WHITE: Apparently they’re people that have been around 

longer. 

DR. LEGREE: That’s not always a good thing. 

DR. LEN WHITE: It’s just interesting. 

DR. LEGREE: But it’s a good point. 


329 




Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Session IV: Assessment 


DISCUSSION 


PARTICIPANT: I’m referring back in my notes to Karl Heider’s definition of 
culture: “Learned shared ideas about behaviors”. So my question is, that I get 
this contrasted with MH shared ideas about behaviors. Anyway my question is: 
is the consensus actually culture--is that what you’re capturing there, and if it is 
does this suggest with this type of measurement there is even more importance 
in somehow norming you’re scoring key when you’re administering the measure 
in a new culture? 

DR. LEGREE: I suppose for some applications that could generally be 
true can you really talk about a driving culture though? Can you talk about a 
culture revolving around G? I’m not sure it’s a sensible notion then, but surely for 
some applications that produces a reasonable set of questions... I’m not sure it 
covers all applications. 
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Performance and Biological Mapping of 
Emotional Awareness 

Professor of Psychiatry, Psychology 
and Neuroscience 

The University of Arizona - 


PERFORMANCE AND BIOLOGICAL MAPPING OF EMOTIONAL 
AWARENESS 

DR. ROBERTS: Our final speaker for today is Professor Richard Lane 
from the University of Arizona. 

DR. LANE: A while ago we had the seventh inning stretch, since I’m the 
ninth speaker I guess that means we’re in the top of the ninth. I want to thank 
the organizers for inviting me, and especially Rich to seeing to it that I got here. 
And it’s really been an extremely informative experience for me and I’ve really 
enjoyed being here. 

Let’s just take a minute and tell you a little bit about my background and to 
help explain why I’ve done the work that I’ve done. I am first trained as a 
psychiatrist and later got a PhD. in Experimental Psychology and in my early 
days, I became fascinated with the question of how stress leads to and affects 
physical disease and that’s how I got into emotion research and I became 
fascinated with the question trying to understand individual differences and the 
response to stress and how emotions were processed. And I went on to get the 
PhD. in part cause I’m very interested in psychometrics and interested in mind 
brain relationships and the brain imaging work that I’ve done has really been an 
effort to understand the physiology of emotion both centrally and peripherally and 
as it mediates between mental events on the one hand and physiological change 
and disease processes on the other. 
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Performance and Biological Mapping of 
Emotional Awareness 


• Levels of Emotional Awareness 


• Neural Substrates of Implicit and Explicit 
Emotional Processes 


• Promoting emotional intelligence 


Here is a basic outline on the talk, the first part major part is going to be on 
levels of emotional awareness which you’ve heard a little bit about, and I’ll talk 
about the theory as well as the scale and measurement. The second part we’ll 
talk about neural substrates of implicit and explicit emotional processes and I’ll 
try to tie it a little bit to the first part as well. And if there is time, implications of 
the model and the findings for promoting emotional intelligence. 
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Emotional Awareness: 

Emotional intelligence consists of a set of mental abilities: 

• Perceive emotion 

• Understand emotion 

• Manage emotion 

• Using emotion 

All require conscious processing of emotional information. 

Emotional int e llig e nce: The ability to use emotional 
information (originating within oneself or the outside 
world) in a constructive and adaptive manner. 


So one of the advantages of powerpoint is that you can change the slides 
up to the last minute and this definition was put on this slide about a half hour 
ago. But I think that emotional awareness is probably fundamental to emotional 
intelligence because we consider these different mental abilities that make up 
emotional intelligence, perceiving emotion, understanding emotion, managing 
emotion and using emotion and all do require conscious processing of emotional 
information. 

Work on emotional awareness, I think, is kind of the foundation or 
substratum of emotional intelligence. One way of defending emotional 
intelligence from this perspective, is that it is the ability to use emotional 
information in a constructive and adaptive manner, with that information referring 
to information coming from within oneself via a kind of introspective process, as 
well as information coming from the outside world. 
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Another perspective on this is that by becoming consciously aware we 
have a lot that intervenes psychologically between the stimulus and response. If 
a car is coming at you and you have that immediate withdrawal response that is 
you know something that we evolved a long time ago and is a very early 
response, an older response and it’s very automatic, and there’s not much that 
intervenes in between these two. 

But here with conscious processing if you’re able to experience your 
emotions you can reappraise what the stimulus is before reaching a final 
conclusion about what the stimulus is; you can think about what your feelings are 
in the setting that you’re in, and perhaps use that information to also think about 
how other people might feel if you were to respond in a particular way. So that 
with conscious processing really amounts to a lot of intervening cognition in 
between stimulus and response and in general it also corresponds to the extent 
of the neural network that mediates between sensory and motor processes. 
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So starting in on the levels of emotional awareness model I actually 
started working on the theory and formulated its basic outline 20 years ago. The 
first paper was published in February 1987 in the American Journal of Psychiatry. 
I am thankful to Gary Schwartz, my first mentor, who helped me create this 
model, which we called the Levels of Emotional Awareness cognitive 
developmental theory. In this first publication, we were also interested in its 
application to psychopathology. 
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Levels of Emotional Awareness: 

The structural characteristics that Piaget 
described for the stages of cognitive development 
also apply to the developmental transformations 
that occur in the capacity to be aware of one’s 
own feelings. 


This was a purely theoretical paper, and the basic idea was that the 
structural characteristics that Piaget described to stages of cognitive 
development also applied to the developmental transformations that occur in the 
capacity to be aware of ones own feelings. 
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The Levels of Emotional 
Awareness Construct 


The higher the level of emotional 
awareness, the greater the differentiation, 
integration and desomatization of 
emotional experience and expression. 


In addition, the model incorporates the idea that the higher the level of 
emotional awareness the greater the differentiation, integration, and 
desomatization of emotional experience and expression. 
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Wine Tasting 


Compared to novices, expert wine tasters: 

• Used more descriptors of wine 

• Used more dimensions in their ratings 

• Accurately rank ordered wines for 
sweetness, balance and tannin (novice^ 
accurately ranked sweetness only) 


One way of understanding this perspective is to consider an analogy and 
that analogy is to wine tasting. A paper published in 1990 by Solomon and 
colleagues looked at experts and novice wine tasters, and they found that 
compared to novices expert wine tasters used more descriptors of wine and they 
used dimensions in the rating and they accurately ranked ordered wine for 
sweetness, balance and tannin, where novices actually ranked sweetness only, 
just one dimension. 

You can think of that in one of two ways, one way would be people that 
who know more have more terms to describe the entity in this case wine and it’s 
just a reflection of their knowledge. An alternative is that the reason they know 
more is that they’ve gone through a complex process of pairing the stimulus with 
verbal descriptors and over time have developed a more complex or complex 
schema or complex mapping of the world of wine. And so the test of that would 
be if you have a novice and expert sip a fine wine and ask them if the taste is the 
same. And how many people think that the taste would be more complex and 
differentiated for the experts? Okay, how many people would say the opposite, 
that it tastes the same? 
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Language, Concepts and Attention 

Language facilitates the development 
of concepts, which influence how 
attention is allocated when confronted 


We’re adopting the view that there’s an intimate relationship between 
language and experience. Warner and Caplan were the ones that put forward 
this view first; that language facilitates the development of concepts which 
influence how attention is allocated when confronted with unfamiliar stimuli. I’ll 
tell you a little bit more about conceptual organization later and also come back 
to the question of attention when we get to some of the brain imaging findings. 
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Representational Redescription 

• The development of knowledge in any domain 
proceeds through a process called “representational 
redescription.” 

• Cognitive development consists of the transformation 
of knowledge from implicit (procedural, sensori-motor) 
to explicit (conscious thought) r ep r esentations th r ough 
the use of language or other semiotic mode. 

• This transformation renders thought more flexible, 
adaptable and creative. 

• The development of intelligence, whether cognitive 
or emotional, is governed by the same principles. 


Now you will appreciate from the preceding slides that I put a lot of focus 
on Piaget and I’m sure many of you are aware that Piaget has come under some 
criticism and I think it is also important to point out that modern conceptions of 
cognitive development are consistent with the point of view that I’m putting 
forward and an example of that is model of Anette Karmiloff-Smith who is a 
protege of Piaget. 

Her view is that the development of knowledge in any domain proceeds 
through a process called representational re-description. Cognitive development 
consists of the transformation of knowledge from implicit procedural and sensory 
motor to explicit conscious thought representations through the use of language 
or other semiotic mode. This transformation renders thoughts more flexible, 
adaptable and creative so the development of intelligence whether cognitive or 
emotional governed by the same principles. 


340 





Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Session IV: Assessment 


Levels of Emotional Awareness 


1: bodily sensations 


2: action tendencies 
3: unidimensional emotional experience 


4: multidimensional emotional experience (blends) 


5: multidimensional experience of self and other 


So we describe 5 levels of emotional awareness patterned after Piaget’s 
stages of cognitive development. But we consider levels 1 and 2 to be implicit 
and levels 3 through 5 to be explicit. Level one is bodily sensations, level 2 
action tendencies, level 3 through 5 involve feelings, level 3 would be a 
unidimensional emotional experience such as the feeling of happiness or 
sadness, a level 4 experience would be multidimensional emotional experience, 
a blended experience like feeling angry and frustrated, and level 5 is 
multidimensional experience of self and other where you have complexity of 
blends in self and other that are different from each other. 
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I’ll show you an example of this, this is a schematic diagram of the levels 
intended to illustrate several points, one is what you see in yellow is implicit and 
what I mean by implicit is that these are visual motor and sensorimotor 
responses, but they are part of an emotional response, but they only could truly 
be considered an emotion if they’re associated with a feeling. And we’ll talk more 
about manifestations of emotion that are not such as a patient who is 
preoccupied with somatic sensations. 
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Conceptual Organization of Emotion 


• Each emotion becomes a conceptual entity in its own right: 

* What the experience is like; how it feels in the body 

• What the outward signs of the emotion are 

• What causes the feeling 

• What enhances or diminishes the feeling 

♦ How the emotional state is related to overt behavior 

♦ How social context influences what is or should be — 

expressed 

• Over time, more features become part of each concept 

• A network of inter-related schemata develops 


So it is a hierarchical model and as you go up the levels each new level 
subsumes the previous ones. The model can be used either in a trait or state 
conceptualization. We have data supporting the trait model; I should tell you that 
the state interpretation hasn’t yet been adequately tested; the idea is any given 
emotional experience would be a micro genetic construction going up the levels 
to the highest level that the person is at, at the moment. So the experience of 
happiness would include visual activation and action tendency of approach and 
unidimensional feeling and the fact that it is consciously experienced would lead 
to a different kind of visual and sensorimotor activation than if it hadn’t been 
consciously experienced. Then a trait interpretation is the typical level that the 
person is functioning at. 
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Conceptual Organization of Emotion 


The conceptual organization of emotion determines: 


• The complexity of subjective experience 

• The ability to recognize emotions 

• The range of emotional expressions 
•The extent of one’s emotion vocabulary 

• The capacity for empathy = 

• The capacity for adaptive social behavior 

• The capacity for emotion self-regulation 


A little bit more theory, and then we’ll get into measurement issues. A few 
years ago I established a collaborative relationship with a woman from Geneva, 
Switzerland who got her Ph.D. with the Piaget group, and she thought that the 
initial theoretical papers didn’t take it far enough, so we’ve written a chapter that’s 
in the handout that talks about this extension of a theory, which is we’re talking 
about the conceptual organization of emotion, which emotional experience, 
emotional awareness is a part. 

So that each emotion becomes a conceptual entity in it’s own right 
consisting what the experience is like and how it feels in the body, what the 
outward sign of the emotion are, what causes the feeling, what enhances or 
diminishes the feeling, how the emotional state is related to a work behavior, how 
social context influences what is or should be expressed. So it’s everything you 
know about an emotion. 

Over time more features become part of each concept and a network of 
interrelated schema developed. And so the theory says that the conceptual 
organization of emotion determines a number of things, the complexity of 
subjective experience, which is what the levels of emotional awareness model is 
about. The ability to recognize emotions, the range of emotional expressions, 
the extent of ones emotion vocabulary, the capacity for empathy, the capacity for 
depth of social behavior and the capacity for emotional social relations. 
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Levels of Emotional Awareness Scale 

• A paper and pencil performance measure 

• 20 emotion-evoking scenes, each described in 2-4 
sentences, each involving two people 

How would you feel? How would the other person feel? 

• Subjects can write as much or as little as needed to 
answer the two questions 

• Each of the 20 scenes is scored separately on a 0-5 scale 


All right! Well how much of this is true? We have a scale called Levels of 
Emotional Awareness Scale to begin to look at this. And I do want to point out 
that you know we really haven’t changed the scale much since we first thought it 
up 20 years ago, and I think it’s in some ways remarkable that it’s worked 
reasonably well, but I don’t think it should be considered a finished product. And I 
think there are other ways of kind of operationalizing the theory even with the 
data that we’ve already collected. 

So it’s a paper and pencil performance measure consisting of 20 emotion- 
evoking scenes, each described in two to four sentences, each involving two 
people. For each scene, two questions are asked. How would you feel? And how 
would the other person feel? Subjects can write as much or as little as needed to 
answer to the questions and each of the 20 scenes is scored separately on a 
zero to five-point scale. Thus, you have a maximum score of 100 with the 20-item 
version. So the reliability is very high, as you’ll see, and it involves picking out 
terms that might be emotional or are emotional and using the glossary to see at 
what level they’re at. So it’s all theory driven. 
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Examples of Emotion 

0: Alone, disbelief, puzzled, skeptical, unsure 
1: Dizzy, exhausted, hot, sick, worn out 
2: Good, bad, upset, rattled, stressed, excellent 
3: Afraid, disgust, happy, love, shame 


Level 1 terms are somatic, so they say, “I feel dizzy. I feel exhausted. I 
feel sick.” These are Level 1 terms. Level 2 corresponds to terms that are used 
more commonly in a non-emotional setting, but are commonly used, often used 
to refer to emotions such as, “I’d feel good,” and “I’d feel bad.” Then Level 3 
discrete emotion terms, unequivocally emotional. 


346 





Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Session IV: Assessment 


Scoring of the 



2: Action tendencies or non-specific terms 
3: One specific emotion term 

4: Two or more specific emotion terms (self or other) 
5: Both self and other are at level 4 and non-identical 


Since we have to account for every word, we come up with the convention 
that words that are cognitive, and aren’t necessarily emotional, are given a score 
of zero, so using the term, “I’d feel puzzled” to be the equivalent of a thought. So 
scoring for each individual item then goes on to Level 4 if there are two or more 
specific emotion terms applied to the self or the other, and Level 5 if both self and 
other at Level 4 and they are non-identical. 
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Scene 20 from the LEAS 


“You and your best friend are in the same line of work. 
There is a prize given annually to the best performance of 
the year. The two of you work hard to win the prize. One 
night the winner is announced: your friend. How would 
you feel? How would your friend feel?” 


So here’s an example: Scene 20 from the Levels of Emotional Awareness 
Scale. You and your best friend are in the same line of work. There’s a prize 
given annually to the best performance of the year. The two of you work hard to 
win the prize. One night the winner is announced - your friend! How would you 
feel and how would your friend feel? 
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Examples of Responses on the LEAS 


• Level 0: I don’t work hard to win “prizes.” My friend 

would probably feel that the judges knew 
what they were doing. 

• Level 1: I’d feel sick about it. It’s hard for me to say 

what my friend would feel - it would all 
depend on what our relationship was like and 
what the prize meant to her 


Here’s an actual Level 0 response: “I don’t work hard to win prizes. My 
friend would probably feel that the judges knew what they were doing.” A Level 1 
response: “I’d feel sick about it. It’s hard for me to say what my friend would feel. 
I would all depend on what our relationship was like and what the prize meant to 
her.” 
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Examples of Responses on the LEAS 


• Level 2: I’d probably feel bad about it for a few days, 

and try to figure out what went wrong. I’m 
sure my friend would be feeling really good. 

• Level 3: We would both feel happy. Hey, you can’t 

win ‘em all. 


Level 2: “I’d probably feel bad about it for a few days and try to figure out 
what went wrong. I’m sure my friend would be feeling really good.” So bad and 
good are Level 2 for both self and other. At Level 3: “We would both feel happy. 
Hey, you can’t win them all.” 
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Examples of Responses on the LEAS 


• Level 4: I would feel depressed - the friend in this light 

is just like any other competitor. I would also 
begrudgingly fell happy for my friend, and 
rationalize that the judges had erred. My friend 
would feel very gratified, but would take the 
prize in stride to save the friendship. 

• Level 5: I’d feel disappointed that I didn’t win but glad 

that if someone else did, that person was my 
friend. My friend probably deserved it! My 
friend would feel happy and proud but slightly 
worried that my feelings might be hurt. 


Okay, Level 4: “I would feel depressed. A friend in this light is just like any 
other competitor. I would also begrudgingly feel happy for my friend and 
rationalize that the judges had erred,” so the self is at Level 4 because of the 
depressed and happy. “My friend would feel very gratified, but would take the 
prize in stride to save the relationship.” So the only real emotion term there is 
gratified, so the other is at Level 3. Here’s an example of Level 5: “I’d feel 
disappointed that I didn’t win, but glad that if someone else did that person was 
my friend. My friend probably deserved it.” So disappointed and glad, you’re at 
Level 4. “My friend would feel happy and proud, but slightly worried that my 
feelings might be hurt.” 
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LEAS Reliability 

%/ 


• Internal consistency (Cronbach’s a) 

• 20-item version (n=371): .84 (S), .83 (O), .88 (T) 

• 10-item versions (n=375): .78(A) .79(B) 

• Inter-rater 

• ICC: r = .84 (1990); r = .96 (1996) 

• Test-retest 

• 2-3 weeks (n=135): .67 (S), .60 (O), .66 (T) (SB) 

• 10 months (n=33): r = .39, p <.03 


So the LEAS has strong psychometric characteristics. It’s really very 
highly reliable. The internal consistency as you see here is in the high 80s and 
the 10-item version, which is just splitting the 20-item version into two halves, 
also has very decent internal consistency. Inter-rater reliability is extremely high 
because there is essentially no inference involved. And test-retest reliability in 
two or three weeks .6 in the Spearman Brown is okay. And at 10 months in a 
small sample it was .39, and we don’t know what that means. It may be because 
the LEAS is sensitive to state effects. 
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LEAS: Psychometric Findings 

• Verbal Ability 

• Other Cognitive-Developmental Measures 

• Openness to Experience 

• Emotion Recognition Accuracy 

• Impulse Control 

• Sex Differences 

• Age and SES effects 

• Discriminant validity 

• Affect Intensity 

• Self-reported Negative Affect 


There have been a number of findings, both from my own group as well as 
from other people, you know well over 100 people have requested and are using 
the scale, but I just want to briefly summarize some of the findings. So there is a 
positive correlation with verbal ability. It ranges from .17 to .38. 

There was a recent paper published in The Journal of Individual 
Differences and Personality and Individual Differences and the correlation there 
with the measure of vocabulary was .27 in a sample of about 100 people. 
Correlates with other cognitive developmental measures, such as Levenger’s 
Sentence Completion Test of Ego Development: the correlations are kind of in 
the mid-20s; with Blatt’s(?) Measure of the Cognitive Complexity of the 
Description of Parents, about the same. LEAS correlates with openness to 
experience in several studies. Emotion recognition accuracy: if you score higher 
on the LEAS you really are more accurate in emotion recognition. 

We did a study of 380 individuals in a community-based sample equally 
stratified for the two sexes, five age groups and three socioeconomic classes, 
and the correlation with emotion recognition accuracy with different kinds of tasks 
was about .43. It correlates positively with impulse control. In practically every 
single study that’s been done women score higher than men, even when you 
control for verbal ability and that’s work that I’ve done with Lisa Feldman Barrett. 
Discriminate validity, it consistently does not correlate with self-reported negative 
affect, and also does not correlate with affect intensity, so these are two major 
dimensions of emotion - valence and arousal, it doesn’t correlate with them, so 
it’s doing something else. 
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LEAS: Recent Psychometric Findings 


Positive correlation between LEAS and Empathy 
(standard self-report measure) 

• Positive correlations between LEAS and MSCEIT 

subscales A (faces), C (blends), D (progression) 

• Predicts likelihood of seeking help for emotional 

problems 

• Correlates with actual amount of social suppo rt 

• Judgements of general well-being not influenced by 

transient mood induction in high LEAS subjects 


Some more recent findings include positive correlations with self-report 
measures of empathy. Now its correlation with the MSCEIT: Kim Barchard 
included the LEAS and the MSCEIT in her dissertation, about 300 people, and 
there were low level correlations that were significant with these subtests with the 
MSCEIT about .18. LEAS also predicted the likelihood of seeking help for 
emotional problems and correlates with actual amount of social support that 
people have. And in this experimental study that was just published in 
Personality and Individual Differences, they did an experimental study where 
they induced either positive or negative mood the between-subject study, and 
they had people rate their general sense of well-being before the mood induction 
and then after. What they found was that people who were low in emotional 
awareness changed their general rating of emotional satisfaction based on the 
mood that they were in, whereas the higher emotional awareness folks did not 
change. 
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LEAS: Clinical Findings 

• Lower scores in Borderline Personality Disorder (Levine et al. 1997) 

• Inverse correlation with non-specific somatic symptoms 
in fibromyalgia 

• Inverse correlation with pain severity in patients with 
irritable bowel syndrome 

• Inve r se cor r elation with intrusive thoughts in patients 

with prostate cancer 6 months after Pennebaker writing exercise 

• Psychosomatic Inpatient Ward - Cologne 


And then there are some clinical findings. Lowest scores on the LEAS are 
for people with borderline personality disorder and there is an inverse correlation 
with nonspecific somatic symptoms and fibromyalgia. An investigator in Syracuse 
told me this week that in a sample of 50 Irritable Bowel Syndrome patients that 
their pain severity was inversely correlated with the LEAS. A recent finding at a 
meeting in Holland is that there’s an inverse correlation with intrusive thoughts in 
patients with prostrate cancer six months after a pen-and-paper writing exercise- 
the more emotional awareness expressed in their writing, the better they did in 
terms of PTSD symptoms. 
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German Psychosomatic Ward 
Inpatient Study 



Susanne Binder, Ph.D. 
Walter Thomas, Ph.D. 
Karl Kohle, M.D. 


And then I want to briefly tell you about an inpatient study on a German 
psychosomatic ward where they have patients, lots of patients with somatoform 
disorders. This involved consecutive inpatients admitted to a ward, well two 
different wards of two general hospitals over a three-year period in Cologne, 
Germany. 
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German Psychosomatic Ward 
Inpatient Study 

* Consecutive inpatients admitted to the Psychosomatic 
Wards of 2 general hospitals over a 3-year period in Cologne 

• Diagnostic groups: 1) depression; 2) anxiety and OCD; 

3) adjustment disorders; 4) somatoform disorders; 

5) psychological factors with somatic disorders; 

6) eating disorders; 249 patients completed treatment 

* TAS - 20, LEAS (10 - item), STAI, SCL - 90 administered at 
onset and end of treatment (2-3 months)^ 

• Multi-modal treatment: psychodynamically-oriented 
individual and group psychotherapy, body-related therapies, 
art therapy, music therapy, and medication as needed 


There were six different diagnostic groups. They admitted those with 
depression, anxiety and OCD, adjustment disorders, somatoform disorders, 
psychological factors associated with a somatic condition, and then eating 
disorders. There were 249 patients who completed treatment. The main point of 
this is that there are advantages of LEAS relative to the Toronto Alexithymia 
Scale in a clinical setting. The TAS-20, as you know, is a self-report measure. 
This is, the LEAS is more difficult to administer and score. Is there a pay-off? I 
think these data will show you that there is. Patients had measures of negative 
anxiety - I’m sorry, negative affect, anxiety, and SCL-90. And the patients 
received multimodal treatment, which was a psycho-dynamically oriented 
individual and group psychotherapy, body-related therapies, art therapy and 
music therapy and medication as needed. The TAS-20 correlated highly, or 
significantly, with measures of negative affect both at onset and at the end of 
treatment. 
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LEAS generally did not correlate with negative affect. Across the six 
diagnostic groups there were no differences between the somatoform patients 
and the other patients with mental disorders in TAS-20 at onset of treatment. 
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Change in TAS-20 With Treatment 


DEPANX ADJ SOM PFSED 



n=54 

n=20 

11=31 

n=68n=38 

n=38 

TAS-20 






Onset 

56.67 (11.8) 

54.41 (13.0) 

55.03 (12.1) 

52.69(11.9) 54.97(11.4) 

55.64(10.3) 

End 

50.69(11.2) 

51.16(12.2) 

52.18(11.1) 

50.98(12.2) 51.45 (11.2) 

51.30(11.4) 


p<.001 NS NS NS p<.05 NS 


In all groups the TAS-20 score went down. In other words, it appeared that 
alexithymia improved. However, when you adjusted for negative affect, these 
changes were not significant. 
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Change in LEAS with Treatment 


Treatment Completers (n=249) 


LEAS 

Onset 

DEP 

n=54 

27.29 (4.6") 

ANX 

n=20 

28.9 (3.9") 

ADJ 

n=31 

29.47 (6.4") 

SOM 

n=68 

25.86 (6.5") 

PFS 

n=38 

26.24 (5.6") 

ED 

n=38 

31.86 (5.2") 



27 66 (7 2 ") 

28 01 (5 2 ") 

29 96 (4 7 ) 

27 7 (7 1") 

28 33 (6 5 ) 

32 57 (7 0 ) 


- ^ ! •\/V7 1 / »I- 

- \ r m n F 1-1 > 7 . L* J - 


^' • ' \ 1 • 1 / 





NS 

NS 

NS 

p<.05 

p<.05 

NS 


With the LEAS there was a difference at onset of treatment where the 
somatoform patients and those with psychological factors affecting physical 
condition had lower LEAS scores than in the patients with mental disorders, and 
this is using the 10-item version. And then within the somatoform groups, there 
was a significant increase from beginning to end of treatment, and when you 
partialed out negative affect, these findings were still significant. 
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So I think now we’re ready for the bottom of the ninth. Because of time 
limitations, I’m not going to be able to give as thorough an explanation, present 
as many findings to support the model that I’m going to present, but the basic 
idea is that we can distinguish between neural substrates of implicit and explicit 
emotional processes. And I think that the evidence is there to support the idea 
that there are three different aspects of conscious awareness of emotion that can 
be distinguished, that have distinct neural substrates, and those are what I’m 
calling reflective awareness, focal attention to feelings and background feelings. 
I’m going to start off with focal attention to feelings because that was a study that 
involved the Levels of Emotional Awareness Scale. 
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Explicit Emotional Processing: Highlighted Structures 



1 - Background Feelings - 

2 - Focal Attention to Feelings 

3 - Reflective Awareness of Feelings 


And just as a preview, we’re going to be focusing on focal attention to 
feelings on this area, which is the dorsal anterior cingulate cortex, which is an 
area that has a super-eminent role in the regulation of attention. This area three 
here, superior medial prefrontal cortex, is involved in reflective awareness of 
feelings, and that’s an area identified by Frith and Frith of being involved in 
mentalizing. And then we have this ventromedial prefrontal area involved in 
background feelings, which I’ll describe for you in a few minutes. 
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I wish to turn now to discuss focal attention to feelings. 
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PET Study of Emotion 
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We did a PET study of emotion. We collected the data about 10 years ago 
now, and we induced happiness, sadness and disgust, as well as neutral control 
conditions in two different ways - with film clips and recall personal experiences. 
We wanted to have comparable conditions for the two kinds of ways of eliciting 
emotion. When we had people recall experiences of sadness, for example, we 
wanted them to focus in on a specific aspect of the memory where the feeling of 
sadness was particularly intense, like focus on that moment when the casket was 
being lowered into the grave, and we told them focus on feeling sad. Then we did 
the injection and we did a one-minute scan. 
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Correlation: Level of Emotional Awareness 
and Blood Flow Attributable to Emotion 


Saggital Coronal 





Coordinates = 14, 6, 30 (BA 24) 
Z = 3.74, pc.OOl 


So to have it be comparable for the film clips, we told them, “Okay, this is 
going to be a happy film. Here’s the background information.” And then we 
showed the film clip and did the scan as they were watching the film. The point is 
that we were having them focus on the particular target emotion. And what we 
found was that when we basically lumped the three emotion conditions together, 
subtracted out the neutral to control for non-specific factors, so we were 
identifying brain activity specifically attributable to emotion, and correlated that 
with LEAS in these 12 healthy women, there was only one area that showed a 
significant correlation and that was in this dorsal anterior cingulate area. 
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Positive Correlation: 



Finding : Higher scores on the LEAS are associated 
with greater activity in the dorsal anterior cingulate 
cortex during emotional arousal. 


Interpretation : The dorsal ACC has a superordinate role 
in the executive control of attention and motor responses. 

The dorsal ACC is activated in response to a variety of 
cognitive and emotional stimuli. Those higher is emotional 
awareness may be better able to attend to their own 
emotional experiences, promoting further emotional development. 


So the finding is higher scores on the LEAS are associated with greater 
activity in the dorsal anterior cingulate cortex during emotional arousal. What 
does that mean? Well, this is our interpretation, that the dorsal anterior cingulate 
has a super-eminent role in the executive control of attention and motor 
responses. The dorsal anterior cingulate is activated in response to a variety of 
cognitive and emotional stimuli, such as pain or even mental effort. The idea is 
that those higher in emotional awareness are activating this area because they 
may be better able to attent to their own emotional experiences and therefore 
engage attentional mechanisms when emotion is aroused. And to the extent that 
they do that, that will then promote further emotional development. This was 
published in The Journal of Cognitive Neuroscience in 1998. There are a variety 
of other findings that are consistent with that, but I’m going to have to move on in 
the interest of time. 
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Now, I would like to turn to reflective awareness and issues surrounding 
this concept. 
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We created another paradigm to try to capture conscious experience of 
emotion, and we used International Affective Picture System pictures, and we 
had people focus on the pictures in two different ways. In half the scans we had 
them focus on the subjective experience that was induced by the pictures. They 
had a three-key keypad and they had to press the key as soon as the picture 
came up. Did it induce a pleasant feeling, an unpleasant feeling or a neutral 
feeling? And the other half of the scans in this PET study we focused on spatial 
location. Was this an indoor scene, an outdoor scene or indeterminate? Now this 
wasn’t simply focal attention because they’re also labeling it with the keypad 
response, so that’s why we think it’s revealing neural substrates of reflective 
awareness. 
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So here are examples of IPS pictures. The picture on the right hand side 
is neutral. 
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And we got very clear results and highly significant results showing really 
highly significant activity here in the medial prefrontal cortex in the peri-cingulate 
region when people were focusing on their internal experience. Well, just like the 
finding with emotional awareness in the dorsal anterior cingulate cortex, it’s not 
an emotion-specific area, but rather an attention area. What we’re finding here is 
that we’ve got an activation in this paradigm because we’re recruiting an area 
involved in a more general cognitive process, that is a mentalizing. 
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C. Frith and U. Frith 


Interacting Minds - 
A Biological Basis 

Science 1999; 

286: 1692-169 5 
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Frith and Frith say that this is a skill that’s developed in people and in 
higher apes to understand the intentions of other animals or other people. And 
that the ability to self monitor comes first from the ability to need to know what’s 
going on in other people’s minds, because there are evolutionary advantages to 
knowing that, whether you’re being deceived for example. 
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I’d like to change focus now and discuss background feelings. 
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Background Feelings 


• Consist of the on-line experience of feelings 

in the absence of attending to them or reflecting 
upon them 

• Add emotional tone to experience but are 

not noticed unless attended to 

• Provide information about the general state of 

well-being - in the background of cs. experience 

• Bias decision-making and behavioral response to 

the environmental “cause” of the emotion 


This is a term introduced by Damasio and I have a somewhat different 
interpretation of it, a little shifting of the use of the term. This is how I use it. It 
consists of the online experience of feelings in the absence of attending to them 
or reflecting upon them. It adds emotional tone to experience, but background 
feelings are not noticed unless attended to. It provides information about the 
general state of well-being, but it’s in the background of conscious experience. It 
may be related, certainly overlaps with Lisa Feldman Barrett’s concept of core 
affect, and it biases decision making and behavioral responses to the 
environmental cause of the emotion. 
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So the work by Damasio and his colleagues on ventral medial prefrontal 
cortex, the somatic marker hypothesis, the people who have lesions here are not 
able to use their gut feeling in making decisions is one example ... 
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D escartes’ 
Mirror 


Emotion, Reason, 
and the Human Brain 

Antonio R. Damasio 


Antonio Damasio 


Joy, Sorrow, 

and the Feeling Brain 


And I certainly recommend these books to you by Damasio. There are a 
number of other areas that are involved in background feelings. 
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A nearby area to the one that I showed before is - we’re calling pregenual 
inter-cingulate cortex. I’m just going to show you an example of a paradigm that 
activates this region. 
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Examples of Individual Trials 

Neutral 


cushion 

cushion 


correct response = 2 


Negative 


murder 

murder 

murder 


correct response = 3 


Example of Blocked Presentation 



Time (minutes) 



This is the emotional counting stroop, so the task involves counting the 
number of words. So a neutral control condition would be words that are neutral 
in meaning, like cushion. The correct response is two. And this, with the word 
murder, would be the negative emotional condition - you have to say three. The 
point is that figuring out how many words there are is in the foreground. That’s 
what you’re focusing on. The emotion is more in the background. 
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And when performing the emotional counting stroop, we’re getting activity 
here in pregenual inter-cingulate. 
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Brain Structures That Participate 

• Ventromedial prefrontal cortex 

• Pregenual anterior cingulate cortex 

• Insula 


These are some of the other structures involved in background feelings. 
For those of you who are familiar with Damasio’s work, you’ll see that these are 
areas that he talks about as being involved in feeling, and I’m saying this is 
background feeling, and in order to - if you actually focused your attention on 
your feelings, then you recruit a different area, and if then you reflect up them. 
You label them and otherwise process them, yet another area is involved. 


379 





Proceedings from the ETS & ARI Emotional Intelligence Workshop 

Session IV: Assessment 



What about implicit emotion? 
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Well, the distinction between implicit and explicit processes is very 
fundamental in cognitive neuroscience and it was I think first best described in 
terms of memory, where we have declarative or explicit memory for facts and 
events, the ability to consciously recall these things. And this is an important 
distinction because the neural substrates for the two processes are distinct. 
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So for explicit memory we have the hippocampus, and hippocampal area 
and the diencephalon on the one hand. And then for implicit memory, skills and 
habits, priming, simple classical conditioning and non-associative learning, each 
is associated with a different kind of neural substrate, so that in anephric 
patients, for example, can be taught a task and they won’t remember that they’ve 
been taught. They won’t know that they know it, but then they can do it. So that 
kind of procedural knowledge, it’s implicit memory. 

And in fact, it’s with the recognition that there are implicit and explicit 
cognitive processes that the boundaries between emotion and cognition I think 
has pretty much disappeared. In terms of the neural substrates of these visceral 
motor and somato-motor components of emotion that I’m calling implicit emotion, 
Cannon and Barge showed in the 1920s that if you made a lesion here at A, you 
eliminated all emotional reflexes and behavior, but if you made a lesion at B, 
keeping parts of the thalamus and hypothalamus intact, you still had emotional 
reflexes such as sham rage, so the display of rage in response to innocuous 
stimulant. 
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And we’ve heard about the low road and the high road in other talks. This 
slide is from Ledoux’s book on the amygdala. The book is called The Emotional 
Brain, but it’s mostly focused on the amygdala. And so what this shows is that 
this man is in the forest. He encounters what might be a coil of rope or it might be 
a snake. And so information goes to lateral geniculate and the thalamus. And 
then there is a low road pathway immediately to the amygdala, and there’s this 
kind of crude representation. It might be a rope. It might be a snake. There’s an 
advantage to thinking and having a response as if it might be a snake, it might be 
inaccurate. That happens out of awareness and it needs to happen out of 
awareness because you need to respond very quickly. Then there’s the high 
road where you can see exactly what kind of snake it is, etcetera. The amygdala 
sends out command patterns to organize autonomic and somato-motor 
responses. It participates in a wider network that does that. 
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Right Amygdala (y = -6; see Figure 2) 


Left Amygdala/S! (y = 0; see Figure 3) 


Some of the first evidence that the amygdala can operate implicitly was 
presented in 1998, in the Journal of Neuroscience. Paul Waling showed that the 
amygdala was activated in response to briefly-presented angry faces that were 
followed by a backward mask consisting of a neutral face. All people saw was the 
neutral face, but when the angry faces were presented very briefly subliminally, 
the amygdala was activated because it’s a threat detector. 
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So to kind of summarize and show parallels between psychological and 
the neuroanatomical, you have a hierarchical neuroanatomical model here and 
what I’d argue is that there’s broad correspondences between what you see in 
yellow here and what you see in yellow there. The amygdala and other limbic 
structures perform their functions outside of conscious awareness, and that their 
output has to go to higher centers to paralimbic structures such as the anterior 
cingulate cortex, orbital frontal cortex and anterior temporal pole as well as the 
prefrontal, superior medial prefrontal cortex for the conscious experience of 
emotion. 
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SERIES IN AFFECTIVE SCIENCE 

Cognitive 
Neuroscience 
of Emotion 



Edited by 

RICHARD D. LANE 
LYNN NADEL 


This model and the justification for it is elaborated in this book that I co¬ 
edited with Lynn Nadel. It’s available in paperback. I don’t get any royalties, but 
it’s published by Oxford University Press in 2000. 
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Implicit Emotion and 

Implicit emotion: the automatic motor 
expressions of emotion that get modulated by 
emotion regulation strategies, including 
autonomic, neuroendocrine, and 
somatomotor {gestures, facial expressions, 
action tendencies, procedures} responses. 


So implicit emotion then is the automatic motor expressions of emotion 
that get modulated by emotion regulation strategies, including autonomic 
neuroendocrine and somato-motor responses, the somato-motor responses 
including gestures, facial expressions, action tendencies and procedures or 
scripts. 
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Emotional Awareness and the Brain: 



1. The fundamental distinction between implicit and explicit 


processes applies to emotion as well as cognition. 

2. Emotion generation occurs without awareness of feelings. 

3. The neural substrates of three types of conscious experience 
can be identified: 

a. Background feelings 

b. Focal attention to feelings 

c . Reflective awareness of feeling s 

4. The neural substrates of conscious emotional experience 
are not unique to emotion but are shared with other 
cognitive functions, setting the stage for individual differences. 


Here are the main points regarding the neural substrates component of 
the talk. The fundamental distinction between implicit and explicit processes 
applies to emotion as well as cognition. Emotion generation occurs without 
awareness of feeling. The neural substrates of three types of conscious 
experience can be identified - background feelings, focal attention, reflective 
awareness - and the neural substrates of conscious emotional experience are 
not unique to emotion, but are shared with other cognitive functions, setting the 
stage for individual differences. So that’s a particularly important point. 
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What are the implications of this model for promoting emotional 
intelligence? You can divide it into normative context and then context of 
maladaptive behavior. 
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Conceptual Organization of Emotion: 

A Psycho-Educational Approach 

• Each emotion becomes a conceptual entity in its own right: 

* What the experience is like; how it feels in the body 

• What the outward signs of the emotion are 

• What causes the feeling 

• What enhances or diminishes the feeling 

♦ How the emotional state is related to overt behavior 

♦ How social context influences what is or should be — 

expressed 

• Over time, more features become part of each concept 

• A network of inter-related schemata develops 


It occurred to me that this conceptual organization of emotion could be a 
guide to promoting emotional awareness in a kind of psycho-educational 
approach, so teaching of people about emotions individually - what the 
experience is like, how it feels in the body, what the outward signs are, etcetera. I 
don’t know about the extent to which such didactic programs have already been 
created, but perhaps it might work. And I think it’s important, as a psychiatrist I 
have a sensitivity to the fact that psycho-educational approaches will work in 
quite a few people, but then there are some people who are going to be resistant 
to change and aren’t going to be able to pick it up that readily. 
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Changing Behavior Patterns 
to Improve Emotional Intelligence 

* Learning to drive a stick shift: with practice, automatic 
procedures eliminate the need for conscious processing 

• Maladaptive behavior patterns that people want or 
need to change (e.g. the “insensitive manager”) 
can be conceptualized as automatic emotional 


And so I think that it’s useful to think in terms of procedures and how 
maladaptive behavior patterns that people want or need to change can be 
conceptualized as automatic emotional procedures. An example was given 
yesterday of the insensitive manager. And so that’s a general behavior pattern. 
Well how do you go about changing that? I think that it’s important to point out 
that in terms of thinking about procedures, learning to drive a stick shift that it first 
requires a lot of conscious processing, then it becomes automatic. And it’s useful 
for these things to become automatic because it eliminates the need for 
conscious processing. It’s very economical. Well how do you undo it? Well I think 
that you have to go from implicit to explicit processing. 
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Emotional Procedures 



• Obtain love and reassurance 


* Handle anger 

* Joke around 

* Resolve conflict 


Emotional procedures are rule-based schemas for knowing how to do 
things related to emotion - express love, obtain love and reassurance, handle 
anger, get attention, joke around, resolve conflict, etc. This kind of approach has 
been written about in the clinical literature and I’ll try to tell you what this guy 
Robert Klinenman said in 1991. 
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Procedural Organization of Emotions 

Clyman R. B. J Am Psychoanal Assn 39: 349-382, 1991 

• During emotional arousal component emotional procedures 

(facial, gestural, overt behavioral) are activated 

• Emotional procedures are repeated in different contexts 

• Families “select” which procedures are permissable and 
which are not 

• Procedures reflect how self and other are expected to behave 


During emotional arousal, component emotional procedures are activated, 
these procedures including facial, gestural, overt behavioral expressions of 
emotion. These emotional procedures are repeated in different contexts and we 
select which procedures are permissible and which are not. Procedures reflect 
how self and other are expected to behave. These procedures develop implicitly 
and with practice are executed automatically. 
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Changing Emotional Procedures 


• Emotional procedures d e velop implicitly and, with 

practice, are executed automatically 

• Once established, procedures are resistant to change 

• Procedures preserve cognitive resources for conscious processing 

• Maladaptive behavior patterns can be conceptualized as emotional 

procedures 

• In circumstances in which simple instruction in alternative procedures 

is insufficient, changing the behavior requires f i rst making the 
i mplicit emotions associated with the procedures explicit — 
(controlled), then developing and practicing new procedures - 

• Making new procedures automatic can take a lot of practice 


Once established, they’re resistant to change. Procedures preserve 
cognitive resources for conscious processing. And we can understand 
maladaptive behavior patterns in this way. In circumstances in which simple 
instructions and alternative procedures are insufficient, changing the behavior 
requires first making the implicit emotions associated with the procedures explicit 
and controlled, and probably in order to do that you have to stop the behavior 
first and ask, interrogate what the feelings are associated with that action 
tendency. And then developing and practicing new procedures and making new 
procedures automatic can take a lot of practice. And I’m a golfer, and one of the 
rules of thumb that I’ve heard is that if you want to change your golf swing it 
takes about 2,000 practice swings to get it automatic. 
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Conclusions 

• The skills that constitute emotional intelligence require 

conscious processing of feelings and exteroceptive 
emotion cues 

• A cognitive-developmental approach can be used to 

understand individual differences in the capacity 
to experience emotion and recognize emotional 

• Cognitive development consists of the transformation of 

implicit into explicit representations 

• This transformation has a neuroanatomical basis 


Okay, so conclusions. 
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Conclusions 


• Implicit and explicit emotional processes have 

distinct neural substrates 

• The neural substrates of conscious emotional experience 

are not unique to emotion but are shared with other 
cognitive functions 

• This helps to explain the need for emotional 

- procedures, and the existence of vast individual 

differences in emotional intelligence 

• Promoting emotional intelligence may be facilitated 

by focusing on shifts from one level to the next 
within a cognitive-developmental framework 


The skills that constitute emotional intelligence require conscious 
processing of feelings and exteroceptive emotion cues. A cognitive development 
approach can be used to understand individual differences in the capacity to 
experience emotion and recognize emotional states in others. Cognitive 
development consists of the transformation of implicit into explicit 
representations, and this transformation I think has a neuroanatomical basis, 
because implicit and explicit emotional processes have distinct neural substrates. 
The neural substrates of conscious emotional experience are not unique to 
emotion, but are shared with other cognitive functions. This helps to explain the 
need for emotional procedures as well as the existence of vast individual 
differences in emotional intelligence. Promoting emotional intelligence may be 
facilitated by focusing on shifts from one level to the next within a cognitive 
developmental framework. Thank you. 
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DISCUSSION 

DR. GADE: Maybe this is a naive, probably is a naive question. Are you 
saying that emotional responses are just like any other automatic response 
except they’re different by the, you can differentiate them by where they occur in 
the brain? 

DR. LANE: Just like any other automatic response? 

DR. GADE: Once you’ve got an automatic response in place, you’re not 
thinking about it or feeling it, like you were talking about driving a stick shift car 
and doing that automatically, and stopping for a traffic light, for example. I 
wouldn’t call that an emotional response. I’d call that a learned automatic 
response. And you differentiate these based on the areas of the brain that get 
activated? Is that it? 

DR. LANE: Yeah. I think that’s what I’m saying, that I think that children 
learn how to negotiate their interpersonal world by learning how to do things: 

How to interact with people, how to express affection, how to resolve conflict, 
etcetera. And I think once a particular interpersonal situation is construed in a 
particular way, that automatic procedures do get set off. Is that; what do you think 
about that? 
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DR. GADE: One is more of a learned response and the other one you’re 
saying is automatically wired in. 

DR. LANE: Oh, no, no. No, I’m not saying it’s wired in. 

DR. GADE: Emotional response is not wired in? You think that’s [cross¬ 
talk]? 

DR. LANE: Let me clarify. The emotional response is kind of part of our 
genetic endowment. But I think that these procedures have an emotion 
regulatory function and they kind of keep emotions in check. It’s a way of 
managing emotions. 

DR. GADE: But you don’t; you’ve already started the behavior in many 
ways and that’s what you’re saying. We’ve already started the emotional 
behavior before you become aware that this action is taking place. 

DR. LANE: That’s right. 

PARTICIPANT: And the difficult part is how do you then delay this kind of 
response on a neurological basis? Do you see that kind of thing happening? 
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DR. LANE: Yeah, well I mean I think Paul Ekman’s discussion of these 
Buddhist monks is very compelling, and I think what it; I mean we do know that 
the frontal lobes have direct projects to these limbic and brain stem structures, so 
I would be willing to bet that that’s how it works, that they’re just very highly 
sensitive to interceptive cues and have massive downward inhibition to regulate 
the evolution of the emotional response. 

PARTICIPANT: I couldn’t follow a lot of your discussion because 
neuropsychology is not my forte. But I was wondering if there is any research on 
people who are dwelling too much on their emotions and how that impacts them. 
Because I can see the benefits of focusing on your emotions, but it also seems 
like there would be certain people that you would want them to be less explicitly 
focusing on their emotions. 

DR. LANE: Absolutely! So one of the things; I mean what immediately 
comes to mind is recent work on rumination. I mean the field is really expanding 
greatly and in the past year there’s been a fMRI study of rumination. And what it 
shows is that rumination is associated with first decreased frontal lobe activity, 
and secondly increased and prolonged amygdala activity. So I mean another 
issue that comes up in response to that is that women are consistently, you 
know, they score higher on the LEAS, and I think in general are more aware. But 
we also know that women have twice the rate of major depression and various 
anxiety disorders than men do. And when is it an advantage? When does it 
become a liability? I think issues such as rumination become relevant and there 
must be some kind of dysregulation that occurs if you’re too focused on your 
feelings. 
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DR. GRANDEY: I was just thinking about the ideas of implicit processes 
and you talked about being aware of your emotions based on your facial 
expressions. I wondered if you could just say your opinion or your view on the 
facial feedback hypothesis. Do you know what I’m referring? 

DR. LANE: Yes. Oh, sure. My opinion about the facial feedback 
hypothesis is that there’s a real effect there, but that it; and I would say that 
Damasio’s work has really been influential in getting me to think that there really 
is something important about bodily feedback. But I think that it’s only part of the 
story, and I think it’s probably a relatively minor effect, although I do know; I’m 
familiar with the studies that say that if you put your face in a particular 
configuration without knowing that it’s emotional that it will be associated with 
certain subjective states and you will get EEG changes that are consistent with 
the affective state that you’re trying to create. I think there’s something to it, but I 
think it’s just one part of a larger picture. 

DR. GRANDEY: But the bodily, and I might’ve missed some stuff too, but 
being aware of your bodily changes was one of the first stages of your 
awareness model. 

DR. LANE: That’s true. 
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DR. GRANDEY: And it’s an implicit process you’re not aware of - correct? 

DR. LANE: Yes. 

PARTICIPANT: So that’s more of the facial feedback. 

DR. LANE: Yes, that is. 

PARTICIPANT: So your model goes beyond...? 

DR. LANE: I think there’s something more than just; I mean Damasio 
would say a feeling is your conscious awareness of your bodily state, and I think 
that there may be something more to it than that, but I think that that’s part of it 
certainly. And I would; well, who am I to disagree with Damasio? 

DR. GRANDEY: I have my own problems with it. Maybe we could talk 
later. I don’t need to carry on. 

DR. LANE: Okay. But I think what you’re implying is that the facial 
feedback hypothesis is entirely consistent with my model. 

DR. JOHNSON: So how do you place in your levels emotions with 
complex cognitive structures such as guilt, remorse, dread, things like that? 
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DR. LANE: Good, no make that an excellent, question. We do not 
differentiate between those kinds of emotions that you just described and simpler 
emotions like happy, sad, anger, fear, love, hate. And that’s an example of an 
aspect of the scoring system that we haven’t addressed, and it would be nice to 
go back and see whether if we somehow took those more complex emotions into 
account, whether that would add to our predictive power of other things. One of 
the things that I want to be sure to say is that Kim Barchard, who’s been 
mentioned several times, who did her dissertation on emotional intelligence, has 
developed a computerized scoring system for Levels of Emotional Awareness 
Scale, and she says that in their most advanced algorithm that the correlation 
with hand scoring is .85. I think when we have a computerized data base and can 
interrogate the data efficiently we can address questions like that. 
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DR. MATTHEWS: I have a question concerning automaticity, because I 
think this is a construct that is easily misused and especially learned 
automaticity. And I think the stick shift example which you gave is constructive 
here, because traffic psychologists - I dabble in traffic psychology - have 
actually debated this issue. And it seems that stick shifting, even among highly 
experienced drivers, is not fully automatic. It requires some cognitive resources. 
And part of the reason for that is that that behavior has to be sensitive to the 
motivational context, because in general you shift down when you’re slowing 
down. But occasionally you want to pass a vehicle, or you want to burst pass, so 
you shift down to do that. It’s serving a different goal. And perhaps the same 
thing applies to emotions. For example, if you’re expressing anger there are 
different tactics which you may be using depending on the context. You might 
want to express anger to show empathy with a friend who’s frustrated about 
something. Or you might want to express anger more in order to express 
aggression to someone. So it’s important that response doesn’t get fully 
automated, because it has to be sensitive to these contextual factors and to 
personal motivations. 
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DR. LANE: And I think what you’re talking about is the difference between 
someone who’s highly emotionally intelligent and someone who isn’t. I think 
somebody who’s at a lower level of emotional intelligence or emotional 
awareness isn’t going to be able to shift gears and recognize that the situation is 
different and change their automatic behavior. And I think the reason why mental 
health clinics are filled with lots of people is that they have maladaptive behavior 
patterns that they don’t know what to do with. And I think what you’re describing 
is somebody who really can go in and out of, and can monitor their automatic 
behavior and switch out of it if need be. 

DR. ROBERTS: Thank you very much, Richard. As you can see we’ve 
finished the innings and are now off to dinner. 


Editors Comments 

This is the end of Volume II. Proceedings are continued in Volume III. 


404 




